PaperlayerView exact revision on arXiv

Source-linked AI summary

Defensive Sufficiency in a Stackelberg Model of AI Security

Subhabrata Majumdar, Rajlakshmi Chavan

arXiv:2610.09892v2cs.CRcs.AI

TL;DR

Generative AI’s expanded, context-dependent attack surface raises questions about when feedback-driven discovery and repair provide sufficient protection and how much defenders should invest. The paper analyzes coverage conditions and a defender-led investment game, finding that finite surfaces can be covered almost surely under stated conditions while faster repair can shorten compromise duration without preventing compromise.

  • Problem

    Generative AI systems have an expanded attack surface, and eventual protection of each attack on a growing surface need not mean complete protection at any finite time.

    Source: §1 · pp. 1–2

  • Method

    The paper models feedback-driven coverage and a defender-led Stackelberg game in which proactive discovery and reactive repair investments precede the attacker’s choice of search effort.

    Source: §1 · p. 2

  • Results

    Under stated conditions, a finite attack surface receives almost-sure complete coverage, while faster repair can reduce compromise duration without reducing compromise probability.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Eventual protection against each fixed attack does not by itself establish negligible current risk on a growing attack surface.

    Source: §1 · p. 2

    The model leaves for future work a repeated game with learned and uncertain capabilities, imperfect retention, attacker adaptation, and benign-task constraints.

    Source: §10 · p. 25

Abstract

from arXiv · show

Feedback from automated testing, human red teaming, and incident response can strengthen an AI system's defenses when discovered failures lead to effective repairs. We study when this feedback process provides sufficient protection and when investing in it is economically worthwhile. We begin by showing that an attack surface composed of finite number of inputs is defended with probability 1 if every unresolved attack has a persistent chance of discovery, repairs are effective, and subsequent updates preserve earlier protection. We derive completion-time bounds and extend the analysis to growing attack surfaces, repairs that generalize across related attacks, and multiple discovery mechanisms. These results distinguish eventual protection against each fixed attack from complete protection at a single time. We then formulate a defender-led Stackelberg game in which the defender invests in proactive discovery and reactive repair, anticipating the attacker's choice of search effort. We characterize the least-cost allocation that deters attack and the equilibrium regimes in which the defender funds neither capability, one capability, or both. Numerical experiments illustrate these regimes and show how faster repair can reduce compromise duration without reducing compromise probability.

1 Introduction

Generative AI systems expose defenders to a broad, context-dependent attack surface, raising the question of when continued discovery and repair provide sufficient protection. The paper analyzes coverage guarantees and a defender’s investment in discovery and repair against an adaptive attacker.

1 Introduction

  • The paper asks when feedback from testing, red teaming, and incident response can provide sufficient protection as discovered failures are repaired.Repairs may fail to cover related prompts, and later system changes can weaken earlier protection.Source: §1 · p. 2
  • Almost-sure complete coverage of a finite attack surface follows under conditions requiring persistent discovery, effective repairs, and retention of earlier protection.The analysis also extends to growing surfaces, generalizing repairs, and multiple discovery mechanisms.Source: §1 · p. 2
  • On a countably infinite attack surface, every fixed attack may eventually be covered while some attack remains uncovered at every finite time.Thus eventual protection of each attack alone does not establish negligible current risk.Source: §1 · p. 2
  • A defender-led Stackelberg game models investment in proactive discovery and reactive repair before the attacker chooses search effort.The model characterizes investment regimes and the least-cost allocation that deters attack.Source: §1 · p. 2
  • Faster repair can reduce compromise duration even when compromise remains possible.The paper also emphasizes verification mechanisms for continued validity of discovery and repair.Source: §1 · p. 3

2 Related Work

The paper connects its coverage analysis to red teaming, repair, and security-investment research. Its distinctive focus is on separate investments in discovery before compromise and repair afterward, alongside the conditions needed for coverage guarantees.

Finding failures and learning from them

  • Automated red-teaming work studies test generation and coverage, but greater empirical attack diversity does not establish persistent discovery of every unresolved attack.Examples include test-generation studies [15], curiosity-driven exploration [8], and tools garak [3] and PyRIT [14].Source: §2 · p. 3
  • MART and rainbow teaming [18] alternate attack generation and safety fine-tuning, while this paper abstracts updates as effective repair and retention assumptions.Retention needs separate justification because later fine-tuning can disrupt shallow safety alignment [16].Source: §2 · p. 3

Scope of coverage guarantees

  • The paper studies evolving guardrails rather than fixed defenses, while noting that no fixed guardrail set can protect against all possible attacks.Its coverage guarantees therefore depend on specifying the attack domain and permitting defensive updates.Source: §2 · pp. 3–4

Security investment and recovery

  • The paper treats discovery before compromise and repair after compromise as separate investments, models attacker response explicitly, and derives a convex joint investment problem with resource-allocation regimes and least-cost deterrent rates.Source: §2 · p. 4

3 A Model of Testing and Repair

The model represents protection as coverage of attack instances and examines how discovery and updates change the vulnerable set. It distinguishes evolving defenses from static ones and clarifies when finite-surface coverage is compatible with an incomplete checker class.

3 A Model of Testing and Repair

  • An attack instance may be a prompt, conversation, or prompt together with its environment; the active surface contains admitted attacks, and covered attacks are those against which the defense is effective.The model allows effectiveness to mean preventing a specified harmful outcome or reducing its probability below a chosen threshold.Source: §3 · pp. 4–5

3.1 Attack discovery

  • Each round discovers unresolved attacks and updates the defense, with assumptions requiring reported vulnerabilities to be repaired by the next round and earlier protection to be retained.The assumptions concern actual repair and preservation, not merely identifying a failure or storing a regression test.Source: §3.1 · p. 5

3.2 Evolving defenses

  • Feedback can progressively strengthen defenses even when each static defense is incomplete; on an infinite surface, the uncovered attack can change as protection grows.The finite-surface proposition states that if each attack has a finite repair time and coverage is monotone, one finite time covers the whole set.Source: §3.2 · pp. 5–6
  • Complete coverage on a finite set cannot be achieved by updates all belonging to a checker class whose every member leaves an element of that same set uncovered.This does not preclude guarantees on a restricted part of a larger universe or with a different class of defenses.Source: §3.2 · p. 6

4 Complete Coverage of a Finite Attack Surface

For a fixed finite attack surface, persistent discovery of each unresolved attack, effective remediation, and retention imply almost-sure complete coverage. The paper also bounds the tail of completion time, while noting that weak discovery can make coverage slow.

4 Complete Coverage of a Finite Attack Surface

  • Under a persistent chance of discovering every unresolved attack, effective remediation, and retention, complete coverage of a fixed finite surface occurs almost surely.The proof does not require successive rounds to be independent.Source: §4 · pp. 6–7
  • P(T > t) ≤ min{1, N exp(−ϵt)} bounds the probability that repairs are incomplete after t rounds on a surface of size N.The bound yields a high-probability completion guarantee when t is chosen using a target failure probability.Source: §4 · p. 7
  • Even a small ϵ suffices for eventual repair, but may require many rounds; more effective exploration shortens the waiting time.Each fixed context level also has finite completion time when it contains finitely many attacks, even if the complete universe is infinite.Source: §4 · p. 7

5 Coverage under Growing Contexts

As attack surfaces grow, protection can be guaranteed for each admitted attack without implying that the whole surface is covered at one finite time. The analysis extends coverage results to generalized repairs and multiple discovery mechanisms.

5 Coverage under Growing Contexts

  • The growing-context analysis treats attack surfaces as an increasing sequence of finite sets, with the context-length limit determining each set.An attack becomes available at its admission time; for tool-using interactions, attack instances may include conversation, retrieved content, tool outputs, and permissions.Source: §5.1 · p. 8

5.1 Complete coverage

  • Each admitted discoverable attack is eventually covered almost surely under the stated assumptions, and every finite context level has a finite completion time.The full countable surface may still have an uncovered attack at every finite time.Source: §5.1 · pp. 8–9

5.2 Robustness toward similar attacks

  • A repair that clears an unresolved region can protect several related attacks after one discovery; larger regions can reduce expected completion time, though rarely encountered regions may take longer.For N = mr attacks, singleton repairs take expected time NH_N, compared with mH_m for region-level repairs.Source: §5.2 · p. 10

5.3.1 Accumulating opportunities for repair

  • An unresolved attack region is eventually repaired almost surely if its cumulative conditional lower bound on repair probability diverges, even when discovery chances decline over time.Repeated opportunities compensate for imperfect regional repair when discovery and repair jointly have a positive conditional lower bound.Source: §5.2 · p. 10, §5.3.1 · p. 11

5.3.2 Combining the mechanisms

  • With multiple discovery mechanisms, each contributes according to both its chance of finding an attack and how often it is selected; a divergent combined contribution guarantees eventual coverage.For a finite surface, this also yields finite-time completion of the whole surface.Source: §5.3.2 · p. 12

5.3 Several discovery mechanisms

  • When several mechanisms operate simultaneously, dependence between reports must be accounted for; incident reports follow actual failures, so they can prevent recurrence but not guarantee against initial harm.Source: §5.3.2 · pp. 12–13

6 A Stackelberg Model for Strategic Defense

The paper models a defender who commits to discovery and repair investments before an attacker chooses search effort. It characterizes the attacker’s response and the defender’s cost-effective investment regimes, including cases where deterrence is not worthwhile.

6 A Stackelberg Model for Strategic Defense

  • The defender commits to automated testing, human red teaming, and incident response before the attacker chooses search effort; the investments determine proactive discovery and post-compromise repair rates.Testing mechanisms share the proactive discovery role, while incident response shortens exploit duration.Source: §6 · p. 13, §6.1 · p. 13

6.2 The attacker’s best response

  • The attacker has a unique best response, choosing zero search effort when the product of discovery and repair rates reaches the deterrence threshold.Greater attacker value or search productivity makes deterrence harder, while greater search cost makes it easier.Source: §6.2 · pp. 14–15
  • Faster repair lowers the benefit of winning the discovery race and the proactive capability needed for deterrence; compromise probability and expected compromise time are nonincreasing in both defensive rates.Attacker effort may initially rise as proactive discovery improves.Source: §6.2 · p. 15

6.3 The defender’s investment problem

  • A Stackelberg equilibrium exists because the defender’s investment objective is continuous, convex, and attains a minimum, though the optimal allocation need not be unique.Deterrence may cost more than the expected loss it prevents.Source: §6.3 · pp. 15–16

6.4 Choosing between d mechanisms

  • For each funded mechanism, marginal benefit equals marginal cost; an unfunded mechanism’s marginal benefit is no greater than its cost, allowing investment in testing, incident response, both, or neither.Source: §6.4 · p. 17

6.5 Defender’s investment regimes

  • With repair rate fixed, expensive proactive capability leads to accepting baseline residual risk, intermediate costs to added protection while attack continues, and sufficiently low costs to full deterrence.The threshold comparisons also depend on defender loss and attacker search cost, value, and productivity.Source: §6.5 · p. 18

7 Joint Investment in Discovery and Repair

The section separates the least-cost allocation that deters attack from the broader decision about whether deterrence is worth buying. It characterizes when the defender invests in neither capability, one capability, or both.

7.1 The least cost of deterrence

  • The least-cost deterrent meets the threshold mµ = θ, with baseline allocation sufficient at zero incremental cost when m0δ ≥ θ.When both rates exceed their baselines, deterrence cost is D = κ(mD − m0) + η(µD − δ).Source: §7.1 · p. 19
  • High proactive-discovery cost favors faster repair, high repair cost favors discovery, and intermediate costs lead to investment in both capabilities.This least-cost allocation does not by itself establish that deterrence minimizes investment plus expected loss.Source: §7.1 · p. 19

7.2 When each investment regime is optimal

  • The baseline is optimal when κ ≥ b0 and η ≥ e0; if m0δ ≥ θ, no additional investment is optimal.The marginal benefits b and e account for the attacker’s response to changed defensive rates.Source: §7.2 · pp. 19–20
  • Deterrence is optimal only under the stated cost thresholds; otherwise, the defender invests in one or both capabilities while attack continues.Deterrence is feasible with unbounded investment and positive effectiveness, but feasibility does not mean purchasing it is optimal.Source: §7.2 · pp. 20–21
  • When both capabilities receive investment, cheaper repair increases the repair rate and reduces discovery while leaving compromise probability unchanged.Expected compromise time falls as repair becomes faster, even though compromise probability does not change.Source: §7.2 · p. 21

8 Numerical Experiments

The experiments evaluate the predicted investment regimes and compare joint discovery-and-repair investment with a fixed-repair benchmark. They illustrate that response investment can reduce compromise duration and deterrence cost, while these parameter sweeps are illustrative rather than empirically calibrated.

8 Numerical Experiments

  • The analytical optimization agrees with constrained numerical optimization across 300 randomized settings, with largest normalized objective discrepancy below 3 × 10^-11.The normalized parameter settings are illustrative rather than calibrated to empirical attack data.Source: §8 · p. 21

8.1 Experiment 1: investment regimes

  • Across 25,921 equilibria, all five predicted regimes appear; at baseline marginal benefits b0 = 2 and e0 = 6, and either κ ≤ 1 or η ≤ 0.25 makes deterrence optimal.Figure 1 maps expected compromise time per opportunity across proactive and repair costs, with white lines marking investment regimes.Source: §8.1 · p. 22, Figure 1 · p. 22
  • At (κ, η) = (1.5, 2), both capabilities are funded but attack continues, with compromise probability 1/3 and expected compromise time 0.204.Across the grid, expected compromise time ranges from zero to the baseline 0.5 time units per opportunity.Source: §8.1 · p. 22

8.2 Experiment 2: effect of attacker cost on defensive investment

  • At cA = 0.25, joint investment and fixed repair both yield compromise probability 0.58, but expected compromise times are 0.25 and 0.58, respectively.At cA = 0.125, expected compromise time falls from 0.79 to 0.30 and total defender cost from 6.57 to 4.10 under joint optimization.Source: §8.2 · pp. 22–23
  • Both models deter at cA = 0.6, while joint investment costs 2.58 versus 3.40 with fixed repair; once cA ≥ 4, neither adds capability.Faster repair limits prolonged exposure even when attack remains profitable, and the survival curves distinguish fewer compromises from shorter ones.Source: §8.2 · p. 23

9 Towards Verifiable Feedback and Retention

The theoretical guarantees depend on detected vulnerabilities being genuine and repaired, and on earlier repairs remaining effective. The section describes verification obligations for testing, repair, and retention that would connect those assumptions to deployed systems.

9 Towards Verifiable Feedback and Retention

  • The sufficiency conditions assume every detected vulnerability is a true positive that gets repaired and earlier repairs remain effective.Applying the results to a deployed system requires evidence that its testing and update procedures satisfy those conditions.Source: §9 · p. 24
  • A failure report alone does not establish a vulnerability, and calling an update a repair does not establish that the vulnerability was removed.The verification layer should connect testing evidence to claims about the evolving defense.Source: §9 · p. 24
  • For randomized systems, one successful retest is insufficient; a defensive claim needs an acceptable failure probability and supporting evidence.Contextual judgments about harmfulness may require additional adjudication even when other requirements can be checked automatically.Source: §9 · p. 24
  • Retention requires each update to preserve earlier accepted adversarial cases as well as add new repair obligations.Benign cases constrain unnecessary blocking, so blocking every input cannot satisfy the finite verification contract.Source: §9 · pp. 24–25

10 Conclusion

The conclusion frames the work around discovery feasibility and the investment needed for effective defense. It calls for longitudinal evaluations and further study of investment decisions as feedback and attacker behavior evolve.

10 Conclusion

  • The paper investigates both the feasibility of vulnerability discovery mechanisms and the investment needed for them to defend AI systems in deployment.The conclusion identifies studying the full feedback loop in deployed systems as a next step.Source: §10 · p. 25
  • Longitudinal evaluations should track whether discoveries lead to effective repairs, whether repairs generalize, and whether later updates reintroduce failures.The verification-layer implementation is a starting point; versioned repair histories and independently validated test cases could connect theory assumptions to practice.Source: §10 · p. 25
  • A repeated game could model defenders learning attack costs, discovery effectiveness, and repair reliability as attackers adapt to protections.The proposed extension includes uncertain capabilities, imperfect retention, and constraints on benign-task performance.Source: §10 · p. 25
Loading 2610.09892v2…