PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing

Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu, Mohammad Sakib Mahmood, Md. Jakir Hossain, M. F. Mridha

arXiv:2608.21423v1cs.CLcs.AIcs.CRcs.MA

TL;DR

Agentic security inherits failures from composing scanners, workflow engines, and language models designed for different operators. This paper systematizes those failures, derives analytic regularities, and argues for deterministic enforcement around model-driven decisions.

  • Problem

    Agentic security systems combine components designed for different operators, creating operational failures at their interfaces, including authentication failures that omit post-login application surfaces.

  • Method

    The paper evaluates security-tool integrations, models recurring failure modes analytically, and frames enforcement through a deterministic layer surrounding model proposals.

  • Results

    The analysis identifies recurring quantitative regularities, including cascade gains, provider-availability effects, and aggregate budget failures in unattended security pipelines.

  • Takeaways & Limitations

    Scope, budgets, severity, and audit should reside in a small verifiable layer that model outputs cannot bypass.

  • Takeaways & Limitations

    The quantitative claims depend on assumptions including recent-first eviction, Pareto runtimes, and conditionally independent cascade errors, while precision claims are not comparable across systems.

Abstract

from arXiv · show

Agentic security uses large-language-model (LLM) agents to plan, dispatch, and interpret security tools. As these systems move from demonstrations to deployed products, practitioners repeatedly encounter the same operational failures. We systematize these failures through a hands-on evaluation of ten widely used static, dynamic, cloud, orchestration, and AI red-teaming tools for unattended pipelines. We introduce a four-dimensional Integration Friction Index that separates one-time engineering cost from recurring organisational, legal, and maintenance cost. We then derive quantitative regularities that explain recurring failure modes. Modelling an agentic security system as stochastic LLM policies wrapped by a deterministic mediator, we show that long-lived sessions lose resident evidence with phase count, while short-lived sub-agents extend the usable horizon according to the compression ratio between raw evidence and its summary. We show that a two-stage verdict cascade multiplies scorer likelihood ratios, but provides little benefit when scorer errors correlate. We show that treating unevaluable outcomes as attack failures biases downstream measurements toward evasive and severe responses. We formulate planner-versus-worker model routing as a knapsack problem and derive a closed-form execution cap for heavy-tailed tools, eta* = alpha v/c. Finally, we show why scope and budget enforcement cannot be delegated to system prompts: prompts do not constrain what actually executes. Inspectra, our implemented platform, serves as a worked instantiation, with mechanisms labelled shipped, partial, or planned, including those that did not work.

I. INTRODUCTION

Agentic security combines security tools, workflow engines, and language models into autonomous pipelines whose failures arise at their interfaces. This paper systematizes those failures through tool integration and quantitative regularities, while arguing that deterministic controls must enforce scope and budgets.

  • Agentic security composes scanners, workflow engines, and language models to plan, attack, verify, and explain without a human in the loop.
  • The paper addresses an engineering gap: capability evidence exists, but the required properties, reasons, and quantitative behavior of deployed systems remain underdeveloped.
  • Ten static-analysis, dynamic-analysis, cloud-audit, orchestration, and AI red-teaming tools were integrated, scored, and sometimes replaced to expose recurring operational failures.
  • Inspectra is used as a worked example, with mechanisms explicitly labelled shipped, partial, or planned rather than presented as uniformly implemented.
  • Four quantitative regularities model context saturation, cascaded verification, model routing, and heavy-tailed tool runtimes.
  • Scope and budget enforcement must govern what executes, because prompt-only controls fail when agent-read content attempts to override the rules.

II. FOUR CONCERNS, ONE REPORT

Agentic security spans four complementary concerns that a single planning loop can correlate in one report. The paper formalizes agentic control as observation-dependent planning above deterministic mediation, rather than merely adding an LLM to a script.

  • II. FOUR CONCERNS, ONE REPORT: SAST reads source code, DAST exercises running applications, PT chains findings into demonstrated exploits, and AIRT attacks embedded models.
  • II. FOUR CONCERNS, ONE REPORT: A single planning loop can correlate a static sink, dynamic confirmation, and AI-layer weakness, unlike four scanners run back to back.
  • II. FOUR CONCERNS, ONE REPORT: The system is modelled as stochastic LLM policies that map observation histories to invocations or terminal artefacts above deterministic components.
  • II. FOUR CONCERNS, ONE REPORT: Autonomy is measured by α, the fraction of plan uncertainty explained by observations; fixed scripts have α = 0, while fully observation-determined plans attain α = 1.
  • II. FOUR CONCERNS, ONE REPORT: Partitioning vulnerabilities across SAST, DAST, PT, and AIRT shows that omitting a concern caps recall for weaknesses discoverable through it.
  • II. FOUR CONCERNS, ONE REPORT: A pipeline using SAST and DAST but never linking a static sink to dynamic confirmation misses vulnerabilities that exist only through that composition.
  • II. FOUR CONCERNS, ONE REPORT: The reference architecture separates stochastic planning, short-lived evidence-gathering agents, an artefact store, and deterministic mediation.

IV. THE CONTEXT ECONOMY

Long-lived sessions lose resident evidence as phase records fill the effective context window, creating a measurable saturation horizon. Short-lived agents and compressed artefact summaries extend that horizon, while batching handles phases that cannot fit initially.

  • IV. THE CONTEXT ECONOMY: A monolithic session loses evidence when cumulative phase traffic exceeds effective capacity Weff = W − w0 under most-recent-first retention.
  • IV. THE CONTEXT ECONOMY: Resident-phase capacity is bounded by ⌊Weff/δmin⌋, so retention fidelity Φn is squeezed toward zero as phase count grows.
  • IV. THE CONTEXT ECONOMY: Short-lived phase agents that store schema-validated summaries let the orchestrator ingest compressed results, extending the horizon according to ρ = s/δ.
  • IV. THE CONTEXT ECONOMY: Figure 2 predicts retention fidelity over phase count and compression ratio, with a coherence frontier separating retained from evicted histories.
  • IV. THE CONTEXT ECONOMY: E[n⋆] ∼ Weff/µ gives the expected phase at which the first record is evicted when phase costs have finite mean µ.
  • IV. THE CONTEXT ECONOMY: With W = 200,000, w0 = 12,000, and approximately 28,000 tokens per phase, the expected saturation point is under seven phases.

B. Hierarchy Buys Exactly the Compression Ratio

Short-lived phase agents extend the usable context horizon by replacing raw evidence with schema-validated summaries, but batching oversized phases introduces lossy carry-forward.

  • Every phase remains fully resident through ⌊W_eff/s⌋ phases when the orchestrator ingests only summaries capped at s tokens.The history-loss point is pushed out according to the summary-to-raw-evidence compression ratio relative to a monolithic session.
  • The compression ratio ρ = s/δ, rather than window size alone, determines how long an orchestrator retains phase history.Doubling W doubles the horizon, while compressing a 28k-token result into a 1.5k summary multiplies it by nineteen.
  • At λ = 0.85, geometric carry-forward loss removes half of the first batch’s detail by the fifth hop.Each hop retains an independent fraction λ of the salient facts it receives, so retained detail decays as λ^(m−1).
  • Oversized files are reported rather than truncated, while first-fit-decreasing packs the remaining files into batches under a conservative payload capacity.The procedure validates each batch result, emits a diagnostic for invalid output, compresses the carry-forward state, and canonicalises the aggregate.

V. THE TOOL LANDSCAPE AND INTEGRATION FRICTION

Unattended tool integration is governed by engineering, organisational, legal, and maintenance costs rather than scanner quality alone. Authentication and authorisation create recurring boundaries that wrappers and explicit gates must expose.

  • Integration friction: Usability for an unattended pipeline depends on four independently paid friction dimensions: engineering, organisational, legal, and maintenance effort.These costs occur on different timescales and are borne by different organisational functions, so rankings change under re-weighting.
  • Dynamic testing: Authenticated dynamic scanners can silently omit the post-login application surface when session or authentication settings do not align.Browser automation can own login and emit authenticated URLs for downstream scanners, separating discovery from probing.
  • Tool landscape: Table II records tool friction as deng/dorg/dleg/dmnt scores, unattended-pipeline fit, and case-study disposition under uniform IFI weights.The table distinguishes tools usable as-is, tools requiring wrappers or explicit gates, and tools not usable in the default pipeline.
  • Dynamic testing: Multi-factor and step-up authentication remain unsupported boundaries for browser-automation-based discovery and should be declared rather than skipped silently.The pipeline should state which authentication modes it supports.
  • Operational and legal friction: Cloud-posture auditing and network reconnaissance can score low on engineering effort but high overall because credentials, approvals, legal authorisation, and ownership constraints recur.The recommended default is integration behind explicit opt-in gates, without waiting for credentials that have not arrived.

VI. A VERIFICATION CALCULUS FOR AI RED-TEAMING

Automated red-team verdicts become unreliable when noisy pattern matchers operate at low prevalence. Precision is determined by the scorer’s likelihood ratio, not by prompt engineering alone.

  • A verdict mechanism’s sensitivity η and false-positive rate ϕ define Λ = η/ϕ, which determines positive-verdict precision at prevalence π.Bayes’ rule expresses precision as PPV = πη/[πη + (1 − π)ϕ].
  • At π = 0.05, η ≈ 0.9 and ϕ ≈ 0.3 yield PPV ≈ 0.14, so six of every seven reported jailbreaks are spurious.The scorer’s likelihood ratio is Λ = 3, and that property belongs to the scorer rather than the prompt.
  • Pattern-matching scorers fire on benign replies containing affirmative openers or enumerated-list lines, producing false positives against modern models.These scorers operate at roughly η ≈ 0.9 and ϕ ≈ 0.3 in the reported setting.

A. Cascades Multiply Likelihood Ratios

Verification cascades improve noisy red-team verdicts when stages fail independently, but correlated scorers erase that advantage. Unevaluable outcomes require a separate disposition rather than automatic rejection.

  • Cascades: Under conditional independence, a two-stage cascade has sensitivity η1η2, false-positive rate ϕ1ϕ2, and likelihood ratio Λ1Λ2.A positive cascade requires both the first-stage scorer and second-stage judge to fire.
  • Cascades: Precision rises from 0.14 to 0.57 with (η1, ϕ1) = (0.9, 0.3), (η2, ϕ2) = (0.85, 0.1), and π = 0.05.This reduces the volume of findings requiring human triage fourfold, at the cost of one additional model call per first-stage positive.
  • Cascades: When scorer errors correlate, the cascade’s effective likelihood ratio decreases with ρ and reaches 1 at ρ = 1, eliminating its benefit.Different model providers, evidence modalities, or rule-based checks are proposed to preserve meaningful stage diversity.
  • Unscoreable outcomes: Figure 4 models precision over prevalence and likelihood ratio, while its population-flow panel separates unscoreable outcomes from the rejected stream.At π = 0.05, the illustrated independent and correlated cascades are compared with a single heuristic scorer; the values are model predictions, not measurements.
  • Unscoreable outcomes: Unevaluable outcomes should remain UNSCOREABLE rather than being absorbed into rejected verdicts, because blocked, truncated, or hedged responses can disproportionately contain successful attacks.The resulting procedure distinguishes CONFIRMED, REJECTED, and UNSCOREABLE and reports bounded coverage.

VII. COST, ROUTING, AND BUDGETS

Unbounded tool runtimes and ad hoc model assignment create coupled cost and reliability problems. The paper recasts routing as knapsack allocation and motivates explicit ceilings for deep-search tools.

  • Runtime budgets: A six-hour scan resulted from leaving an exploitation tool without a stopping condition.The tool had been allowed to investigate indefinitely.
  • Model routing: Model routing assigns roles to tiers using token volumes, tier quality, and role weights to quantify upgrade gains and costs.
  • Model routing: Routing efficiency θr = ∆qr/∆cr is quality gained per dollar, making two-tier assignment a 0–1 knapsack problem.Multitier routing becomes a multiple-choice knapsack.
  • Model routing: Sorting roles by routing efficiency is optimal for the relaxed problem and remains within one role’s quality gain of the true integral optimum.The guarantee applies to the relaxed allocation and its resulting integral choice.
  • Model routing: A strong-planner, cheap-worker strategy fails when large token volume and quality sensitivity produce only middling routing efficiency.Long-context extraction from messy pages is the cited example.
  • Spend allocation: At price ratio r = 5, holding planner-tier traffic below 37.5% of tokens halves total spend relative to routing everything to that tier.Automatic escalation after cheap-call validation failures can erase the saving.

B. Budgets Are a Reliability Control

Heavy-tailed security-tool runtimes require budgets treated as reliability controls, not merely spending limits. The paper derives a per-call cap and combines it with separate session and orchestrator ceilings.

  • Per-call caps: For Pareto runtime X, α ≤ 1 makes uncapped expected runtime diverge, while capped utility is U(β) = vP[X ≤β]−cE[min(X, β)].Here v is completion value and c is running-time cost per unit.
  • Per-call caps: The utility-maximizing cap is β⋆ = αv/c, depending on the tail shape and the value-to-cost ratio.These quantities are intended to be estimated by practitioners.
  • Per-call caps: At α = 1, expected billed time grows logarithmically with the cap, while completion probability saturates as 1−xm/β.This makes generous caps comparatively inexpensive but eventually yields little additional completion probability.
  • Layered budgets: A per-call cap cannot bound aggregate spend: n calls may consume nβc, and tool-time budgets omit orchestrator token costs.The paper therefore requires separate ceilings outside policy control.
  • Layered budgets: Rule 2 sets β⋆ = αv/c, adds a per-session ceiling, separately caps orchestrator turns, and reserves five to ten per cent for reporting.Budget-truncated classes are labelled BOUNDED-BY-TIME.

VIII. THE ADVERSARIAL ENVIRONMENT

Agentic security faces adversarial targets, attacker-controlled content, and changing provider safeguards. The paper places scope and budget checks in a deterministic mediation layer rather than relying on prompts.

  • Threat fronts: Targets, read content, and model providers create three distinct hostile fronts requiring different remedies.The target is adversarial, content can inject instructions, and provider safety posture can change.
  • Scope enforcement: The mediator evaluates g(a) = I[target(a) ∈D] for every invocation and executes it only when the proposed target lies within authorised scope.The predicate is independent of the policy that proposed the invocation.
  • Scope enforcement: A deterministic scope predicate makes executed scope violations identically zero, including when the policy is fully compromised.A compromised policy can propose invocations but cannot bypass the shared predicate.
  • Prompt limits: Prompt-only scope or budget instructions fail under sampling variance, context saturation, and injected instructions in scanned content.The paper characterizes prompts as requests rather than execution controls.
  • Mediated invocation: Algorithm 3 logs requests, checks scope before budgets and execution, sanitises arguments, applies the minimum applicable deadline, and records bounded coverage.The invocation path returns typed refusals when invariants would be violated.

A. Guardrail Drift, and Why It Is Not the Vendor’s Fault

Provider guardrails can drift independently of a deployed agent, while model stochasticity also destabilizes repeated scans. The paper responds with exposure reduction, canonicalisation, and explicit reproducibility metrics.

  • Guardrail drift: A provider safety-classifier change caused previously accepted authorised adversarial-prompt work to be refused without code or model-version changes.The described mid-2026 sequence included global access suspension and later control changes.
  • Guardrail drift: Threshold changes that catch more misuse also flag more legitimate traffic when legitimate requests lie near the decision boundary.The paper treats this false-refusal trade-off as structural rather than an implementation detail.
  • Mitigations: Recommended mitigations include dated model snapshots, deterministic payload templating, explicit authorised scope, and behavioral canaries before campaigns.These measures reduce exposure to provider changes rather than negotiating with them.
  • Availability: An eighteen-day annual outage implies q ≈0.049: one provider gives about 95.1% availability, while two give roughly 99.76% under optimistic independence.Shared regulatory actions can make independence optimistic.
  • Verdict severity: Cyber Jailbreak Severity scores capability gain, breadth, weaponisation ease, and discoverability on [0, 4], with capability gain gating the final band.Zero capability uplift remains informational regardless of weaponisation ease.
  • Reproducibility: Repeated scans can differ because agreement across L sampled decisions is bounded by κ̄^L, making structural reductions in sampling more effective than temperature tuning alone.Rule-based classification, bounded turns, and schema validation are cited interventions.
  • Reproducibility: Canonicalisation enables semantic stability through content-hash deduplication, stable sorting, lookup-table severity mapping, and set agreement over discovered surfaces.Byte-level replay of one recorded transcript remains an audit measure.

IX. CASE STUDY: INSPECTRA

Inspectra is a customer-hosted agentic penetration-testing platform combining multiple analysis and red-teaming capabilities behind an LLM orchestrator. Its implementation emphasizes durable workflows, deterministic mediation, explicit mechanism status, and practical deployment constraints.

  • Inspectra combines static analysis, authenticated dynamic analysis, and AI red-teaming behind an LLM orchestrator for customer-hosted deployment.
  • Every scan is a durable workflow whose phases provide retries, timers, cancellation, and restart survival beyond an agent SDK alone.Deterministic triage and file mapping precede model calls, while reconnaissance agents run on the planner tier.
  • Shipped mechanisms include hierarchical orchestration, artefact handoff, dual budgets, event logging, deterministic triage, authenticated discovery, and red-team dispatch.The platform also uses deterministic severity mapping and content-level secret scoring.
  • PyRIT’s supported interpreter changes forced a Python 3.12 backend for most of development, while healing required fresh disposable environments and dependency-directory exclusion.These integration constraints show that runtime compatibility, environment isolation, and input hygiene are operational requirements.
  • Managed Kubernetes is the lowest-effort deployment answer for Inspectra’s multi-container stack, with an estimated floor of around $280 per month.The estimate is directional, and provider platform support is subject to retirement and migration timelines.
  • Customer-hosted deployment keeps source code inside the enterprise perimeter but leaves durable protection for vendor logic primarily contractual rather than technical.Air-gapped customers must accept either an egress proxy or the cost of self-hosting open-weights models.

X. DISCUSSION

The discussion frames agentic security as an engineering discipline constrained by analytic assumptions, implementation failures, dependency volatility, and deployment economics. It concludes that deterministic mediation—not model prompting—must govern execution boundaries and operational controls.

  • The paper’s quantitative claims depend on explicit assumptions about eviction order, Pareto runtimes, conditional independence, illustrative operating points, and ordinal friction scoring.Inspectra precision and recall are not reported because its real targets lack labelled ground truth.
  • Provider policy should be treated as a first-class engineering dependency, while broader generalisation from the reported vendor episode remains limited.
  • Identifier drift can cause static and dynamic evidence for one vulnerability to be treated as separate findings, and content hashing mitigates but does not eliminate the problem.
  • The paper proposes target-descriptor standardisation, calibrated verdicts, and formal mediation-layer verification as directions for addressing transfer, scoring, and enforcement gaps.
  • Production constraints outweigh model capability in the reported evidence, especially alpha-quality dependencies, volatile provider safety policy, model-tier costs, and deployment topology costs.
  • Scope and budget enforcement are reference-monitor properties that prompts cannot supply, placing every LLM policy outside the trusted computing base.
  • The defensible architecture lets models propose while deterministic code controls scope, budgets, severity, and audit against attacker-controlled material and real infrastructure.
  • The paper’s practical value is shortening diagnosis of recurring autonomous-system failures from prolonged debugging to focused reading.
Loading 2608.21423v1…