PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling

Mingqian Feng, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Christopher White, Jianfeng Gao

arXiv:2601.22636v2cs.AI

TL;DR

Single-shot and low-budget safety evaluations may miss adversarial risk that emerges under repeated parallel sampling. SABER models heterogeneous per-sample vulnerability with a Beta distribution and derives a scaling law for extrapolating large-N attack success from small measurements. Using n = 100, it predicts ASR@1000 with mean absolute error 1.25 versus 12.04 for the naive baseline, while the paper reports heterogeneous scaling profiles and rapid risk amplification at scale.

  • Problem

    Existing safety benchmarks often use single-shot or low-budget querying, whereas large-scale parallel probing can repeatedly test models and make real-world adversarial risk difficult to assess.

  • Method

    SABER models query-level per-sample success probabilities with a Beta distribution and combines fitted parameters with an analytic scaling law to estimate ASR@N from small-budget measurements.

  • Results

    Using n = 100, SABER predicts ASR@1000 with mean absolute error 1.25 versus 12.04 for the naive baseline, an 89.6% error reduction.

  • Takeaways & Limitations

    Models that appear robust under single-shot evaluation can exhibit rapid risk amplification at scale, supporting scaling-aware safety evaluation beyond ASR@1.

  • Takeaways & Limitations

    The main scaling approximation is intended for the large-N regime; under α < 1 and β < 4, its error is reported below approximately 0.1% for N ≥ 5.

Abstract

from arXiv · show

Large Language Models (LLMs) are typically evaluated for safety under single-shot or low-budget adversarial prompting, which underestimates real-world risk. In practice, attackers can exploit large-scale parallel sampling to repeatedly probe a model until a harmful response is produced. While recent work shows that attack success increases with repeated sampling, principled methods for predicting large-scale adversarial risk remain limited. We propose a scaling-aware Best-of-N estimation of risk, SABER, for modeling jailbreak vulnerability under Best-of-N sampling. We model sample-level success probabilities using a Beta distribution, the conjugate prior of the Bernoulli distribution, and derive an analytic scaling law that enables reliable extrapolation of large-N attack success rates from small-budget measurements. Using only n=100 samples, our anchored estimator predicts ASR@1000 with a mean absolute error of 1.66, compared to 12.04 for the baseline, which is an 86.2% reduction in estimation error. Our results reveal heterogeneous risk scaling profiles and show that models appearing robust under standard evaluation can experience rapid nonlinear risk amplification under parallel adversarial pressure. This work provides a low-cost, scalable methodology for realistic LLM safety assessment. We will release our code and evaluation scripts upon publication to future research.

1. Introduction

Standard low-budget safety evaluations can underestimate risk when attackers use large-scale parallel sampling. SABER models this scaling behavior to extrapolate large-budget attack success from small measurements.

  • Motivation: Large-scale parallel probing can elicit harmful responses after thousands of attempts, making single-shot or low-budget evaluations underestimate adversarial risk.Attack success can rise sharply as sampled prompt augmentations increase, including with simple surface-form variations.
  • Approach: SABER models per-sample success probabilities with a Beta distribution and derives an analytic scaling law linking small-N observations to ASR@N.The Beta model provides a tractable representation of uncertainty in per-sample success rates.
  • Evaluation: The framework evaluates multiple advanced jailbreak pipelines under adversarial conditions intended to resemble realistic operational settings.This broader evaluation characterizes both the presence and behavior of scaling vulnerabilities.
  • Results: SABER predicts large-scale adversarial risk accurately from small-budget measurements.The paper presents this as a low-cost alternative to brute-force evaluation at large sampling budgets.
  • Results: Using n = 100, SABER predicts ASR@1000 with mean absolute error 1.25 versus 12.04 for the naive baseline, an 89.6% error reduction.Attack methods show heterogeneous Sample-ASR distributions and distinct risk-scaling profiles.

2. Preliminaries

The paper defines jailbreak evaluation as an attacker–victim–judge pipeline and measures attack success across harmful queries. Best-of-N sampling generalizes single attempts by repeatedly sampling the pipeline, while Beta–Binomial modeling represents uncertain success probabilities.

  • Jailbreak attack and evaluation metric: A jailbreak pipeline consists of an attacker producing an adversarial prompt, a victim model generating a response, and a judge scoring whether the model is jailbroken.The judge returns a score and determines the jailbreak outcome.
  • Beta–Binomial distribution: The binary outcome convention defines 1 as jailbroken and 0 as not jailbroken.
  • Jailbreak attack and evaluation metric: ASR is the expected proportion of harmful queries on which the attacker successfully jailbreaks the victim model.Without a suffix, ASR usually denotes ASR@1.
  • Scaling with Best-of-N sampling: Best-of-N sampling repeatedly samples adversarial prompts until a harmful response is elicited or N attempts are exhausted.The setting is equivalent to N parallel attempts, producing ASR@N.
  • Scaling with Best-of-N sampling: Each attempt independently samples the full attacker–victim–judge pipeline under the paper’s simplifying assumption.The assumption also covers analogous repeated sampling of victim responses or judges.
  • Beta–Binomial distribution: A Best-of-N attempt is modeled as a Bernoulli trial with unknown success probability θ and a Beta prior θ ∼ Beta(α, β).After k successes in n trials, the posterior is Beta(α + k, β + n − k).

3. Scaling-Aware Best-of-N Estimation of Risk

SABER estimates large-N Best-of-N adversarial risk by fitting a Beta distribution to small-budget, query-level success probabilities and applying an analytic scaling law. The framework accounts for heterogeneous vulnerability, supports practical extrapolation and uncertainty quantification, and clarifies when asymptotic approximations are reliable.

  • Framework: SABER collects n attempts per query, fits a Beta(α, β) distribution to estimated sample success probabilities, and extrapolates ASR@N through an analytic scaling law.The three-stage framework uses a small measurement budget, estimates (α̂, β̂), and predicts large-N adversarial risk.
  • Motivation: The scaling profile is essential because ASR@1 alone cannot characterize effective jailbreaks, while direct large-N evaluation is often prohibitively expensive.Best-of-N sampling makes large-N risk a more operationally relevant measure under automated parallel probing.
  • Scaling law: Large-N risk is governed by heterogeneous query-level success probabilities, with α controlling risk-amplification speed and Γ(α + β)/Γ(β) determining the leading constant.The model assumes query-level probabilities follow a Beta distribution and attempt outcomes are conditionally independent Bernoulli trials.
  • Approximation scope: For common jailbreak pipelines with α < 1 and β < 2, the asymptotic ASR@N approximation has error below ≃1% for N ≥20.A small-N correction is provided for smaller N or parameter settings outside this typical region.
  • Estimation: The Beta–Binomial MLE explicitly models finite-budget sampling uncertainty, avoiding the noise-free treatment of per-query estimates used by the two-stage baseline.Its estimation variance decreases as either the per-query budget n or the number of queries K increases.
  • Operational estimation: The plug-in SABER estimator combines fitted (α̂, β̂) with the scaling law and accommodates query-specific attempt budgets, while confidence intervals follow from MLE covariance estimates.For target N ≤20, the small-N correction can further improve accuracy.

4. Experiments

Experiments evaluate SABER across attackers, victims, judges, budgets, and target sampling scales. The anchored estimator consistently improves large-N risk prediction, including uneven-budget and small-budget settings.

  • Experiment Settings: Experiments span HarmBench, three jailbreak attackers, two victim models, and two judges.The attackers cover stochastic perturbations, learned suffix generation, and strategy-driven rewriting.
  • Main Results: 86.2% relative error reduction: SABER-Anchored lowers ASR@1000 MAE from 12.04 to 1.66 using n = 100 trials per query.The evaluation aggregates attacker–victim–judge combinations on HarmBench.
  • Main Results: At N = 1000, increasing n from 50 to 200 reduces baseline MAE from 14.1 to 5.5 and anchored MAE from 2.3 to 1.3.At fixed n = 100, anchored MAE remains 1.7 at N = 500 and 1.8 at N = 1000, while baseline MAE rises from 7.8 to 9.4.
  • Small-N Prediction: SABER predicts ASR@50 within 0.25% error with n = 10, versus 22.73% for the baseline on one early-saturation triplet.The triplet is Jailbreak-R1 against GPT-4.1-mini with the HarmBench Classifier.
  • Uneven Budgets: Uneven budgets systematically increase MAE, but SABER-Plugin outperforms anchoring at n = 50, N = 200 and reduces baseline error by at least 7.32%.The uneven-budget setting samples each query budget uniformly from 1 to 2n.
  • Uneven Budgets: Reducing the visible query set from K = 159 to K = 80 changes MAE from 1.7 to 2.1, while K = 40 yields MAE 2.9.The method continues to outperform the baseline across all evaluated K values.
  • Uneven Budgets: For target ASR τ = 95%, the method predicts a required budget of approximately 466 versus the ground-truth 484 in one representative setting.This result uses n = 200 measurements under Augmentation against Llama-3.1-8B with the HarmBench Classifier.

5. Conclusion

SABER predicts large-scale adversarial risk from small-budget Best-of-N measurements by modeling per-sample vulnerability with a Beta distribution. The conclusion emphasizes that risk can amplify rapidly with sampling budget, requiring evaluation beyond ASR@1.

  • SABER predicts large-scale adversarial risk under Best-of-N sampling from small-budget measurements.Its scaling behavior is modeled through the per-sample vulnerability distribution.
  • A Beta distribution over per-sample vulnerability yields an analytic scaling law for extrapolating attack success rates.
  • Models appearing robust under single-shot evaluation can exhibit rapid risk amplification at larger sampling budgets.The conclusion also reports that attack approach rankings can reverse as the sampling budget changes.
  • The findings support scaling-aware safety evaluation beyond ASR@1.

B.1.1. PROOFS OF THEOREMS

The proofs derive Best-of-N success probabilities by conditioning on latent per-query success rates and then averaging under a Beta distribution. Gamma-ratio asymptotics provide the large-N approximation used in the scaling law.

  • Gamma-ratio asymptotics are applied with z = N, a = β, and b = α + β to derive the expansion used in Equation (6).Multiplication by Γ(α + β)/Γ(β) produces the claimed expansion.
  • The probability of no success across N attempts is represented by the event that every attempt outcome is zero.
  • Averaging this event over a Beta-distributed latent success probability gives an exact closed form.
  • Under the Beta model, ASR@N is obtained by averaging the OR-aggregated success indicator over harmful queries.
  • Keeping only the leading asymptotic term gives 1 − ASR@N proportional to N^-α with a Gamma-function coefficient.

B.1.2. UNBREAKABLE-SAMPLE MODELING EXTENSION

The extension addresses harmful queries that may be deterministically unbreakable, which violates the continuous Beta-mixture assumption. A spike-and-slab model introduces a zero-success mass so risk saturates below one, while online settings distinguish independent prompts from repeated attempts on one query.

  • The continuous Beta-mixture assumption implies ASR@N approaches 1 as N grows.
  • Some harmful queries may be unbreakable because of hard refusals, safety filters, or constrained decoding.Such queries correspond to non-negligible probability mass at θi = 0.
  • A spike-and-slab prior represents unbreakable queries with a point mass at zero and fraction π.
  • With unbreakable queries, risk saturates below 1 at a level determined by π.
  • The online extension assumes harmful prompts have probability ph and benign prompts always receive a non-jailbroken judge outcome.
  • For repeated attempts on one sampled harmful query, the number of harmful prompts is modeled as Binomial(N, ph), and conditional jailbreak probability follows the scaling law.

B.1.5. ERROR BOUNDS

The appendix develops asymptotic error analyses for the standard and corrected scaling laws and describes likelihood-based estimation of the Beta parameters. It also identifies noise in a two-stage fit and accommodates heterogeneous query budgets.

  • Error bounds: The error analysis bounds truncation error using the first neglected term of the asymptotic expansion.The bound preserves the first neglected term’s sign and scales with α-dependent powers of the expansion variable.
  • Error bounds: Convexity of f(x) = x^-α is used to constrain secant slopes between tangent slopes in the error analysis.
  • Error bounds: The appendix defines residuals for both the standard approximation and the small-N correction to compare their deviations.
  • Parameter estimation: A two-stage fit estimates each query-level success probability as ki/n before fitting a Beta distribution to those estimates.
  • Parameter estimation: When n is small, treating estimated success probabilities as noise-free can produce miscalibrated, overconfident estimates.
  • Parameter estimation: The one-stage Beta–Binomial MLE models sampled outcomes directly and can be optimized with standard constrained methods.
  • Parameter estimation: Heterogeneous query budgets are handled by replacing the common n with each query-specific budget ni in the likelihood terms.

B.2.2. ESTIMATION UNCERTAINTY

The section quantifies estimation uncertainty for the Beta–Binomial MLE and shows that increasing the number of queries K reduces variance more consistently than increasing the per-query budget n. Closed-form asymptotic results agree with Monte Carlo estimates for moderate-to-large K, while small-K discrepancies reflect the limits of asymptotic normality.

  • Uncertainty quantification: The closed-form covariance estimate is obtained by inverting the observed information matrix, enabling confidence intervals for plug-in and anchored ASR@N estimators.The delta method applies to the plug-in estimator, while the anchored interval uses a transformation depending only on α̂.
  • Variance scaling: Increasing K continuously narrows the empirical sampling distributions of α̂ and β̂, including a clear difference between K = 500 and K = 1000.The observed narrowing is consistent with the closed-form variance scaling.
  • Variance scaling: Var(α̂) and Var(β̂) scale inversely with K, with increasing K reducing estimator variance at a K^-1 rate.The scaling follows from Fisher information growing linearly with K when n is fixed.
  • Budget effects: For very large per-query budgets n, the standard deviations of α̂ and β̂ decrease more slowly than under increases in K.This comparison is illustrated in Figure 8.
  • Approximation limits: Closed-form and Monte Carlo estimates agree across moderate-to-large K, but diverge at small K because asymptotic normality is less accurate there.This is an author-identified limitation of the covariance approximation.

C.1.5. IMPLEMENTATION DETAILS.

The evaluation uses repeated sampling and bootstrap subsampling to simulate small-budget observations, then validates the Beta assumption and studies allocation under a fixed total budget. Results indicate that the best allocation depends on the parameter being estimated.

  • Evaluation procedure: Each attacker–victim–judge triplet receives 10,000 independent samples per harmful query to establish ground-truth ASR@N for N ≤ 1000.Small-budget observations are generated by bootstrap sampling n trials without replacement from these samples.
  • Model validation: 9/12 (75.0%) triplets pass the Beta-assumption validation, while three failures show borderline p-values or excess boundary mass.The failing configurations suggest bimodal rather than unimodal heterogeneity.
  • Budget allocation: With fixed per-query budget n, both estimator variances scale as O(1/K), so fixed-budget optimization reduces to selecting n and setting K = B/n.A finite optimal n* yields a query count that scales proportionally with the total budget B.
  • Budget allocation: Under fixed total budget B = nK = 10000 and ground-truth (α, β) = (0.4, 4), α̂ has minimum standard deviation around n = 10 and β̂ around n = 5.These allocations correspond to K = 1000 for α̂ and K = 2000 for β̂.

C.2.3. PREDICTING N FOR A TARGET ASR LEVEL

The section introduces Budget@τ as an inverse risk-scaling metric that estimates how many attempts are needed to reach a target ASR level. It is designed to make attack effort interpretable and cost-aware using small-budget measurements.

  • Metric definition: Budget@τ is the predicted number of attempts required to reach ASR@N = τ, such as Budget@90%.Unlike ASR@1, it explicitly captures Best-of-N scaling behavior.
  • Interpretation: Similar ASR@1 values can correspond to dramatically different required budgets when the fitted amplification parameter α̂ differs.The metric distinguishes risk profiles that single-shot evaluation can treat similarly.
  • Practical use: Budget@τ can be reported from small-budget measurements by fitting α̂ and optionally anchoring the estimate at ASR@n, without brute-force evaluation at very large N.This makes the metric aligned with operational attack costs.

D. Limitation and Future Work

The paper identifies scope limitations in outcome modeling, benchmark coverage, attacker and victim coverage, and residual estimation bias. It proposes extensions to non-binary judgments, broader evaluations, stronger pipelines, and improved calibration.

  • Outcome modeling: The current formulation assumes binary judge outputs, limiting direct treatment of scores or multi-class severity assessments.The proposed extension replaces the Bernoulli–Beta hierarchy with a categorical likelihood and Dirichlet prior.
  • Evaluation scope: Evaluation is limited to the textual HarmBench benchmark, three attacker paradigms, and models that exclude costly frontier safety-aligned systems.Future work targets multimodal tasks, larger datasets, stronger attackers, and a wider range of victims.
  • Estimation bias: Residual SABER error tends to underestimate ASR@N more often than overestimate it, motivating work on bias correction and calibration.The proposed direction aims to preserve extrapolation guarantees while improving calibration.
Loading 2601.22636v2…