PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming

Alexandra Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, Hanna Wallach

arXiv:2601.18076v1cs.LG

TL;DR

The paper addresses when attack success rates can support comparisons of relative system safety or attack-method efficacy, given that many such comparisons use apples-to-oranges estimands or low-validity measurements. It combines measurement theory and inferential statistics to formalize ASRs and identifies conceptual coherence and measurement validity as the key conditions for meaningful comparison, illustrating them through jailbreaking.

  • Problem

    ASR comparisons are routinely used for evaluative claims, but their evidence may involve incongruent estimands or measurements that do not validly represent safety or attack efficacy.

  • Method

    The paper combines social science measurement theory and inferential statistics to model ASRs as estimates of threat-model-defined estimands, using jailbreaking as a running example.

  • Results

    Meaningful ASR comparisons require both conceptual coherence between estimands and measurement validity of the ASRs as measurements of those estimands.

  • Takeaways & Limitations

    Comparative red-teaming studies should define what is being measured, ensure estimands are meaningfully comparable, and examine whether prompts and evaluation procedures validly represent the target concept.

  • Takeaways & Limitations

    The argument most directly applies to red-teaming activities that produce comparable ASRs, not qualitative existence-proof activities intended to surface examples or unknown unknowns.

Abstract

from arXiv · show

We argue that conclusions drawn about relative system safety or attack method efficacy via AI red teaming are often not supported by evidence provided by attack success rate (ASR) comparisons. We show, through conceptual, theoretical, and empirical contributions, that many conclusions are founded on apples-to-oranges comparisons or low-validity measurements. Our arguments are grounded in asking a simple question: When can attack success rates be meaningfully compared? To answer this question, we draw on ideas from social science measurement theory and inferential statistics, which, taken together, provide a conceptual grounding for understanding when numerical values obtained through the quantification of system attributes can be meaningfully compared. Through this lens, we articulate conditions under which ASRs can and cannot be meaningfully compared. Using jailbreaking as a running example, we provide examples and extensive discussion of apples-to-oranges ASR comparisons and measurement validity challenges.

1 Introduction

AI red teaming increasingly uses attack success rates (ASRs) to support comparative claims about system safety and attack-method efficacy. The paper argues that such comparisons require both comparable estimands and valid measurements, rather than relying on observed ASR differences alone.

  • AI red teaming probes generative AI systems for vulnerabilities, memorization, and failures of safety alignment.
  • ASRs quantify the fraction of attacks judged successful and are used to compare systems, mitigations, and attack methods.
  • The paper asks when ASRs can meaningfully reflect relative system safety or attack-method efficacy.
  • Meaningful ASR comparison requires conceptual coherence: the compared population parameters must support the intended evaluative claim.It also requires measurement validity: ASRs must validly measure those population parameters.
  • The paper uses measurement theory and inferential statistics to formalize ASRs as estimates of threat-model-defined population parameters.Jailbreaking serves as the running example for analyzing when comparisons are supported.

2 Background: Descriptive, Inferential, and Evaluative Claims

An observed ASR difference is descriptive, but claims that one jailbreaking method is generally superior are inferential and evaluative. Meaningful evaluation therefore requires both evidence about the relevant population parameters and a defensible account of whether those parameters represent the intended notion of efficacy or safety.

  • Observed ASRs of 0.65 and 0.4 show that one tested jailbreak set was judged successful more often than the other.This is a descriptive claim about the specific attacks, system configurations, outputs, and success determinations.
  • Calling the higher-ASR jailbreak generally superior extends beyond the observed attack outcomes and makes an evaluative claim.
  • Treatment-superiority studies similarly distinguish observed outcomes from inferences about broader population parameters.Observed treatment-group outcomes are used as estimates of population-level survival parameters, with statistical testing addressing sampling evidence.
  • Evaluative conclusions require estimands that meaningfully represent the target concept, not merely statistically distinguishable estimates.For treatments, side effects, costs, and accessibility may matter beyond survival rates.
  • ASR comparisons likewise use observed estimates to support inferential and evaluative claims about relative safety or attack efficacy.
  • Comparable estimands are necessary for meaningful ASR comparisons; comparing different time horizons would be an apples-to-oranges comparison even with statistical significance.The paper argues that confidence intervals and hypothesis tests alone cannot resolve incongruent estimands.

3 Attack Success Rates as Measurements

The paper frames ASRs as measurements produced by an operationalized jailbreaking activity and intended to reflect estimands defined by a probabilistic threat model. This framework separates conceptual gaps from measurement error and makes validity conditions explicit.

  • The framework studies prompting-based jailbreaking against a target genAI system, including system configurations that affect responses.The paper focuses on this common setting while noting that attacks involving model weights or fine-tuning APIs require a different setup.
  • A jailbreak transforms a harmful base prompt, and a judge assigns success when the resulting response meets the attack-success criteria.
  • A jailbreaking method constructs transformations using a construction system, prompts, and a judge, which may differ from the target system or evaluation judge.
  • The measurement framework links background concepts, systematized activity, operationalized activity, and observed ASRs through systematization, operationalization, and execution.ASRs are treated as measurements of estimands defined through a probabilistic threat model.
  • A probabilistic threat model specifies success criteria, a distribution over harmful base prompts, and conditions governing the jailbreaking activity.
  • The operational judge is only an instrument for collecting evidence about the underlying success concept, not the concept itself.The target of inference is the systematized success criterion rather than the particular judge used in an experiment.
  • Measurement validity asks whether the instantiated jailbreaking activity accurately measures the estimands defined by the systematized activity.Accurate measurement cannot compensate for a systematization that fails to represent a meaningful notion of safety or efficacy.

4 Conceptual Coherence: Deconstructing Attack Success Rates

Attack success rates can encode different estimands when studies vary in aggregation, sampling, decoding, or success definitions. The paper shows that comparing such quantities can produce apples-to-oranges conclusions about jailbreak efficacy.

  • Conceptual coherence: ASRs can differ conceptually because success rates may aggregate attack-instance outcomes into goal-level success differently, making success easier by changing the aggregation rule.For jailbreaking, the goal is typically defined by the base harmful prompt, and the aggregation of attack-instance indicators determines goal-level success.
  • GCG versus GE: GCG uses one output per prompt after 500 optimization steps under one fixed configuration, whereas GE samples 392 responses across 49 configurations and selects among them.These procedures therefore entail different estimands before their reported ASRs are compared.
  • GCG versus GE: Replacing the attacker scorer with the judge in GE’s selection phase caused only a modest ASR reduction in the experiments.The paper therefore presents the estimand under the simplifying assumption that the judge was used throughout.
  • GCG versus GE: GE’s Top-1-of-392 ASR measures whether at least one sampled response succeeds, while GCG’s ASR is one-shot, so comparing them is fundamentally apples-to-oranges.For one-shot success probability p0 ≥ 0.01, the probability of at least one success in 392 attempts is at least 0.98.
  • Repeated sampling: Higher temperature can leave average one-shot success unchanged while increasing Top-1 ASR through high-entropy sampling and repeated aggregation.The probability that a prompt succeeds at least once, 1 −(1 −p0)^K, rises rapidly as p0 moves away from zero.
  • Repeated sampling: Repeatedly sampling base prompts at temperature 2.0 produced a Top-1-of-50 ASR of 0.83, close to the reported 0.88(±0.04) for LAA.The authors argue that new jailbreaks evaluated with Top-1-of-K metrics should demonstrate improvement over this repeated-sampling baseline.

5 Measurement Validity

ASR measurement can fail when prompts do not represent the intended harmful-behavior domain or when judges do not validly identify success. These problems can bias comparisons across systems and jailbreak approaches even when evaluation setups are standardized.

  • Harmful prompts: High baseline ASRs can reflect prompts that do not meaningfully measure violations of intended safety policies.Some evaluated prompts appear permissible under reasonable interpretations of provider policies, raising concerns about face and content validity.
  • Harmful prompts: In one political-activities category, only 1 of 10 prompts clearly violated OpenAI policy, while 7 were borderline and 2 were clearly non-violating.
  • Success criteria: Meaningful ASR interpretation requires clearly defined success criteria because undesirable behavior is often not systematized, especially for complex or contested risks.
  • Judge validity: Judges can produce biased ASRs when their true-positive and false-positive rates differ across target systems, even when systems have identical true attack success probabilities.Equal overall judge accuracy or AUC is insufficient because system-specific TPR and FPR can make identical systems appear different in safety.
  • Judge validity: 0.46 and 0.60 are the expected ASRs for two systems with the same true attack success probability of 0.5 when judge error differs across systems.The judge has overall accuracy 0.8 on both systems, yet underestimates one system and overestimates the other.
  • Jailbreak approaches: Judge error can also vary across jailbreak approaches, making a shared judge insufficient for valid comparisons.Certain target-model hallucinations can trigger false-positive success determinations, and output length may be associated with changing observed ASRs through judge error.

6 Discussion

The paper’s validity framework is aimed at quantitative red teaming and also connects to broader genAI evaluations. Its scope excludes activities primarily intended to produce qualitative existence proofs rather than measurements.

  • Scope: The framework most directly applies when AI red teaming is intended to produce quantitatively comparable ASRs.
  • Scope: Some red-teaming activities instead produce qualitative examples of undesirable behavior or surface unknown unknowns without attempting quantification.
  • Broader relevance: The measurement-theory framing can also be applied to other task-oriented genAI evaluations, including evaluations of stereotyping or memorization.
  • Recommendations: When ASRs are intended to support comparisons of system safety or attack efficacy, the activity should be designed for conceptual coherence and measurement validity.

A.1 AI red teaming

AI red teaming is widely used to probe genAI vulnerabilities, but its expanding adoption has outpaced attention to limitations. The paper narrows its detailed analysis to jailbreaking while noting limits to broader generalization.

  • AI red teaming: AI red teaming encompasses methods for probing genAI systems for vulnerabilities and undesirable behaviors.These methods can identify vulnerabilities, detect intellectual-property memorization, and assess safety alignment against harmful-use policies.
  • AI red teaming: Automated approaches increasingly use genAI systems to generate attack inputs and judge whether attacks succeeded.
  • Limitations: Red teaming’s continued growth has occurred without fully addressing known limitations or uncovering unknown ones, complicating what can be learned from it.
  • Scope: The detailed discussion focuses on jailbreaking, although the arguments generalize to many quantitative evaluation approaches under the broader red-teaming umbrella.
  • Scope: The measurement-theory lens is less suited to red teaming focused on producing examples rather than quantifying or systematically characterizing behavior.

A.2 Standardizing and systematizing jailbreaking

Standardizing jailbreak evaluations addresses some experimental variation but does not by itself ensure valid ASR comparisons. The paper frames threat models as part of specifying the quantities being estimated.

  • Standardization: Jailbreaking studies have repeatedly emphasized standardizing potential confounding factors.
  • Standardization: Observed ASR decreases with output length, motivating recommendations to standardize the number of tokens generated by target models.
  • Systematization: Threat models such as black-box, white-box, and transfer settings help specify the estimands underlying jailbreak evaluations.
  • Validity: Standardizing the experimental setup is insufficient when judge TPR and FPR vary across target models or jailbreak methods.

A.3 LLM-as-judge

Meaningful ASR comparison requires aligning what is measured, how success is defined, which attacks are sampled, and how results are aggregated. The paper frames these requirements through measurement theory and inferential statistics.

  • Measurement procedure: LLM judges can materially change jailbreaking metrics, making judge choice part of the measurement procedure.The supplied passages identify LLM-as-judge systems as important for automated refusal evaluation and report that judge choice has a big impact on resulting metrics.
  • Measurement framework: Valid measurement requires a coherent link from the broad concept to a systematized concept, instrument, and instance-level measurement.Measurement theory supplies these four levels for moving from a concept of interest to numerical measurements and for interrogating reliability and validity.
  • Measurement validity: Using the same procedure is neither necessary nor sufficient for valid comparisons across settings.A contaminated benchmark can invalidate comparisons even when the same benchmark procedure is repeated, while different procedures may remain valid when they measure the intended concept appropriately.
  • ASR definitions: ASRs differ by attack sampling, transfer versus in-sample evaluation, and the number and aggregation of attempts counted as successful.These dimensions alter the threat model or success criteria and include Top-1, one-shot, and best variants.
  • ASR definitions: ASR terminology is inconsistent, so identical labels such as Top-1 can denote different estimands and aggregation procedures.The paper contrasts goal-specific success criteria with universal maximum-over-transformations reporting and notes the absence of widely adopted nomenclature.
  • Cross-risk comparisons: Comparisons across risk areas are problematic when prompt difficulty is not principledly calibrated, because the compared ASRs may reflect different goal distributions.The paper distinguishes comparisons grounded in deployed-system interactions from comparisons based on arbitrary dataset heuristics.

D Discussion of Harmful “Political Activites” Prompts from Chu et al. [11]

The discussion finds that many political-activity prompts are underspecified or ask about capabilities rather than requesting prohibited outputs. Consequently, classifying responses as successful violations often depends on policy interpretation, conversational scope, and prompt wording.

  • Overall assessment: 1 of 10 prompts was a clear policy violation, 7 were borderline, and 2 were clearly not violations under reasonable policy interpretations.The authors identify a recurring issue: many prompts ask whether the model can or would do something rather than requesting the prohibited activity itself.
  • Scope boundary: The policy itself has changed since the evaluated study, limiting direct interpretation of its findings under the current policy text.The analysis relies on the policy likely in effect when the earlier study was conducted, while noting that a current version is available.
  • Prompt interpretation: Underspecified prompts can make generic campaign assistance look policy-relevant even when the request does not specify political activity or a target demographic.The discussion says answers not specific to political campaigns should not be classified as violations and distinguishes generic templates from tailored campaign materials.
  • Interaction scope: Single-turn ASR can classify requests as nonviolations even when later turns might elicit policy-violating materials.Several assessments depend on whether the model would ask for missing policy, demographic, or issue information in another turn.
  • Facilitation threshold: Providing high-level instructions is borderline because whether it constitutes facilitating a prohibited activity depends on the policy interpretation and the user’s existing expertise.The discussion repeatedly notes that generic or technically insufficient instructions may not meet the policy’s facilitation threshold.

E.1 Experimental setup

The experiments reproduced prior sampling configurations while adding repeated samples, larger temperatures, and additional prompt/model settings. Observed non-determinism in supposedly greedy decoding produced variable per-prompt attack success probabilities.

  • Implementation behavior: Two implementation changes addressed inference instability and efficiency: switching from float16 to bfloat16 and batching attack configurations across prompts.The experiments ran on 4 A6000 GPUs.
  • Sampling configurations: The study reproduced 49 sampling configurations, collected 49 samples per prompt-configuration-model triple, and added temperatures T = 1.5 and 2.0.The additional temperatures were run across all 100 prompts with 49 samples each.
  • Implementation behavior: Top-k sampling with k = 1 was intended to match greedy decoding, but auto-sharding and bfloat16 produced non-deterministic outputs.This caused some per-prompt success probabilities averaged over 49 iterations to fall between 0 and 1.
  • Implementation behavior: The researchers verified that the non-determinism originated in standard HuggingFace API calls rather than their implementation.The same behavior appeared in a standalone unit test using k = 1, do_sample=True, auto-sharding, and bfloat16.
  • Prompt and model coverage: Experiments also applied the configurations to all 51 prompts from Chu et al. and the Llama-13B-chat model.The same code was used with a different newline-separated prompt file.

E.2 Additional Results

Additional experiments show that configuration changes can leave one-shot ASR relatively stable while altering the distribution of per-prompt success probabilities. Temperature-related trends also differ across Llama models.

  • Figure 4: One-shot ASR changes little across configurations, while increasing temperature greatly increases entropy in the per-prompt attack success probability distribution.Fewer base prompts have a statistically 0% chance of producing undesirable responses at higher temperatures.
  • Figure 4: Top-1 aggregation over repeated sampling produces very high observed ASRs when temperature increases the distribution’s entropy.The figure links higher observed Top-1 ASR to more prompts acquiring nonzero success probability.
  • Figure 5: Llama 2 7B Chat shows a more significant upward trend in one-shot ASR as temperature increases than Llama 2 13B Chat.The comparison is most relevant for the two Chat variants because the base Llama models lack the same safety alignment.
  • Figure 5: The experiments include ASR-versus-configuration results for three models beyond the Llama 2 13B Chat model presented in the main paper.The figure focuses interpretation on the two Chat variants.
Loading 2601.18076v1…