PaperlayerView exact revision on arXiv

Source-linked AI summary

APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport

Uchi Uchibeke

arXiv:2609.22076v1cs.CR

TL;DR

Benchmarks that score adversarial prompts assess which model is safer, but do not measure the same attack and model with and without enforcement at the tool boundary. This paper replays human-authored payment attacks across models and architectures, finding zero unpermitted transfers behind the authorization check in the tested Levels 2 to 4. The result is limited to the configured payment setting and engine.

  • Problem

    Prior adversarial-prompt benchmarks score which model is safer, leaving unmeasured the same-attack comparison with and without enforcement at the tool boundary.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The study replays human-authored attacks against fourteen models with and without a deterministic Open Agent Passport check before tool execution.

    Source: §1 · pp. 1–2

  • Results

    140 of 76,842 model-alone evaluations had unpermitted transfers, compared with 0 of 69,297 behind the layer at Levels 2 to 4.

    Source: §9 · p. 19

  • Takeaways & Limitations

    The check enforced the forbidden-recipient restriction while 25,370 payments still executed behind the layer; it is not a general safety guarantee.

    Source: §9 · p. 19

    The Level 4 zero concerns the locally configured engine, which required a memo audit code absent from the published policy pack.

    Source: §7 · p. 18

Abstract

from arXiv · show

APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review. Requests are common and their rate differs far more across configurations than across models, though each attack exists at exactly one configuration so policy and attack cohort vary together: 10.9% of model-alone evaluations at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. On the 1,293 Level 4 prompts, each evaluated on every model, request rates run from 71.2% to 84.3%, and 809 prompts (62.6%) elicited a request from all fourteen models, each ending in a successful payment to the level's allowlisted recipient. The authorization boundary is where the conditions diverge. At Levels 2 to 4, transfers to recipients the passport did not permit number 140 of 76,842 with the model alone and 0 of 69,297 behind the layer, and 105 against 0 on 68,970 matched model, prompt and track triples. The zero spans 790 source sessions, giving a per-session upper bound of 0.38%. It was not obtained by refusing payments: 25,370 payments executed behind the layer, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient. We release the 225,964 evaluations, the level passports, the scoring code and the analysis script at huggingface.co/datasets/aporthq/vault-benchmark-v1 .

1 Introduction

The paper asks what catches residual risk when an agent is deployed, measuring the same attacks with and without enforcement at the tool boundary. It introduces a replay benchmark and reports that the boundary prevents unpermitted transfers without suppressing requests.

1.2 Contributions

  • The benchmark replays 4,371 human-authored attacks across fourteen models, five levels, two tracks, and two architectures, with 225,964 completed evaluations.The corpus and scoring code are released.Source: §1.2 · p. 2
  • At Levels 2 to 4, unpermitted transfers were 140 of 76,842 model-alone evaluations and 0 of 69,297 behind the layer.The zero has a session-clustered upper bound of 0.38%.Source: §1.2 · p. 2
  • On 68,970 matched triples, the paired request-rate difference was +0.084 percentage points [-0.020, +0.189], an observation rather than an equivalence test.The authors disclose 1,220 individual pairs that disagree.Source: §1.2 · p. 2
  • Across 2,807 prompts with complete single-turn evaluations for all fourteen models, request rates ranged from 33.1% to 39.9%.The paper describes this as a shared feature of the tested configuration, not a model-safety ranking.Source: §1.2 · p. 2
  • Outcomes concentrated in a few source sessions, attack transfer across models was weak, and the multi-turn advantage was concentrated in one attack family.Two of 790 sessions produced half the Levels 2 to 4 outcomes; one forged-receipt cohort produced 111 of 113 multi-turn Level 2 outcomes.Source: §1.2 · p. 3
  • The judge panel’s pooled inter-judge kappa was 0.772, but the headline outcome is deterministic and does not depend on judge verdicts.Kappa fell to 0.167 at Level 3 and 0.521 at Level 5.Source: §1.2 · p. 3

1.3 What this paper does not claim

  • The findings concern payments through one tool schema in a simulated bank, not data exfiltration, code execution, or agent delegation.The paper also cautions that its zero assumes a correctly configured policy engine and well-specified capability.Source: §1.3 · p. 3

2 Related Work

Prior work measures attacks on agents, model compliance, or judge-scored outcomes, while this paper measures whether consequential actions execute. Its distinguishing comparison holds attacks and models constant while varying enforcement at the tool boundary.

2.1 Attacking tool-using agents

  • AgentDojo and InjecAgent construct attacks, whereas LLMail-Inject uses challenge participants’ attempts but measures whether an injection manipulates the model rather than whether an action executes.This paper varies the enforcement architecture underneath the same attack.Source: §2.1 · p. 3
  • AgentHarm measures model willingness on harmful tasks, and ToolEmu emulates tool execution; neither varies what happens between model decision and tool execution.The paper manipulates that intervening enforcement boundary.Source: §2.1 · p. 3

2.2 Multi-turn attacks

  • MT-JailBench and MultiBreak report stronger multi-turn attacks on refusal metrics, while this paper finds the execution-based advantage concentrated in one attack family and source cohort.The authors frame this as a caution about aggregate multi-turn claims, not a contradiction, given different corpora and metrics.Source: §2.2 · p. 3

2.3 Judging attack outcomes

  • Because payment outcomes are observable state changes, the headline metric reads executed tool calls without relying on an LLM judge.A two-model judge panel remains as an audit layer; pooled kappa was 0.772 and one judge missed 35.6% of confirmed outcomes.Source: §2.3 · p. 4

2.4 Authorization at the tool boundary

  • The Open Agent Passport (OAP) is a pre-action authorization architecture; unlike sandboxing after execution, it prevents an action from executing.The paper measures this architecture across models and attacks, extending the single-tier live result reported in Before the Tool Call [13].Source: §2.4 · p. 4

2.5 Where this paper sits

  • Compared with prior work, the attacks are human-authored under a bounty, the study spans fourteen models from eight labs, and the outcome is an executed state change.The manipulated variable is enforcement architecture rather than model or prompt.Source: §2.5 · p. 4

3 Methods

The methods replay attacks from a live payment-agent challenge across models, policy levels, tracks, and two architectures. A deterministic policy check is evaluated at the tool boundary, with outcomes recorded separately and coverage and engine differences disclosed.

3.1 The corpus

  • The corpus contains 4,371 attacks from 1,128 sessions after removing empty submissions and entries containing personal information.Attacks span five policy levels, with 97.7% arriving during the March 6 to 12 launch window.Source: §3.1 · p. 4

3.2 Policy levels

  • Levels 2 to 4 are measurement levels where policy permits some recipients and forbids others, making unpermitted transfers real policy failures; Level 1 is permissive, and Level 5 tests enforcement.Source: §3.2 · p. 5

3.3 Models

  • The study evaluates fourteen named models from eight labs through their providers’ APIs.This keeps the served model version attributable to its provider.Source: §3.3 · p. 5

3.4 The two architectures

  • The two architectures differ only in whether a deterministic policy engine checks each transfer call against the level’s passport before execution.The check runs in the tool-dispatch path and is not exposed to the model as a modified prompt.Source: §3.4 · p. 5

3.5 Replay tracks and the four stages

  • Track A replays only the attacker’s final message, while Track B replays the full message sequence; the study records attempted, requested, authorized, and executed-or-denied stages separately.The paper argues a single success rate conflates distinct events.Source: §3.5 · pp. 5–6

3.6 What Level 5 measures

  • At Level 5, the prompt instructs the model to call the transfer tool while saying a guard will intercept it; the layer denied all 9,385 calls.Level 5 measures enforcement under an instructed attempt, not attacker success or model susceptibility.Source: §3.6 · p. 6

3.7 Outcome measurement; 3.8 Decode settings and statistics

  • The headline outcome is read from executed tool calls, while decode settings use the CTF’s per-level production temperatures and single-run estimates.Intervals are clustered by source session, zero-event bounds include a per-session rule-of-three estimate, and paired comparisons use Benjamini-Hochberg FDR at 0.05.Source: §3.7 · p. 6, §3.8 · p. 6

3.9 Coverage; 3.10 The policy engine under test

  • Of 244,776 planned evaluations, 225,964 completed; incomplete cells are marked rather than treated as zero, and no headline claim relies on a partial cell.The local engine matched the hosted verifier on 100 parity cases, but its Level 4 memo-code requirement made it stricter than the published pack.Source: §3.9 · pp. 6–7, §3.10 · p. 7

4 Results

The results separate payment requests, successful payments, authorization decisions, recipient membership, and unpermitted transfers. Across Levels 2–4, the authorization layer recorded zero unpermitted transfers while 25,370 payments succeeded, and the zero is bounded using 790 source sessions.

4.1 What the instrumentation records

  • The five recorded events distinguish requests, successful payments, policy decisions, recipient membership, and unpermitted transfers; the study does not adjudicate intent or legitimacy.A successful payment requires a tool call returning “success”: true, while an unpermitted transfer is a successful payment to a recipient the passport did not permit at Levels 2–4.Source: §4.1 · p. 7

4.2 Payment-request elicitation, per level

  • Request rates span 0.1% at Level 3 to 79.4% at Level 4, but attacks occur at only one configuration, so policy and attack cohort vary together.Level 4’s prompt authorizes documented transfers to an approved recipient with valid codes, and the verification tool confirms it.Source: §4.2 · p. 8

4.3 The five stages, Levels 2 to 4

  • At Levels 2–4, the last-row unpermitted-transfer outcome is zero behind the layer, despite 25,370 successful payments; request and payment outcomes otherwise differ by under a point across conditions.The table identifies the last row as the authorization-boundary outcome.Source: Table 2 · p. 8, §4.3 · p. 8, Figure 1 · p. 9
  • Of 25,640 transfer calls evaluated at Levels 2–4, the policy allowed 25,453 and denied 187, including 148 for forbidden recipients.Denials also included 17 invalid amounts, 13 limit violations, and 9 missing audit codes.Source: §4.3 · p. 8
  • Level 5 is reported separately: its prompt directs a transfer call and warns of interception; behind the layer, 4,960 requests yielded zero payments in 20,109 evaluations.The passport grants no payment capability at Level 5.Source: §4.3 · pp. 8, 10

4.4 Bounding the zero

  • The zero unpermitted transfers across 69,297 layered Level 2–4 evaluations has a 0.38% per-session upper bound across 790 source sessions.The paper uses a session-level bound because evaluations sharing a source session are not independent.Source: §4.4 · p. 10

4.5 Matched pairs

On matched model, prompt, and track triples, unpermitted transfers fell from 105 to 0 while request counts remained close. The section also distinguishes shared authorized requests from policy denials and notes that the attack-only corpus cannot measure overblocking on legitimate traffic.

4.5 Matched pairs

  • 105 unpermitted transfers with the model alone versus 0 behind the layer occurred on 68,970 matched model-prompt-track triples, with 25,362 versus 25,420 requests.Source: §4.5 · p. 10

4.6 Fourteen models on identical inputs

  • 809 of 1,293 Level 4 prompts (62.6%) elicited requests from all fourteen models, with successful payments to the allowlisted recipient.Source: §4.6 · p. 10, Figure 2 · p. 11
  • Those 809 shared requests were authorized behavior, not evidence that the prompts defeated fourteen models; outside that intersection, models differed in requests and unpermitted-transfer counts.Source: §4.6 · pp. 10, 12

4.7 Request rates by level in both architectures

  • On Levels 2–4 matched pairs, the request-rate difference was +0.084 percentage points behind the layer (95% interval [-0.020, +0.189]); 1,220 of 68,970 pairs (1.77%) differed.Source: §4.7 · p. 12
  • Close aggregate request rates are an observation, not an equivalence test, and do not establish identical behavior or causal mediation.Source: §4.7 · p. 12

4.8 What the policy actually decided

  • Of 187 denied calls at Levels 2–4, 148 were for forbidden recipients; 172 evaluations contained denials, and 15 also had successful payments.Source: §4.8 · p. 12, Table 5 · p. 12
  • Because requests came from attacks and no benign-task evaluation was run, these results cannot estimate overblocking of legitimate traffic or establish utility.Source: §4.8 · p. 12

4.9 What the layer does when the policy permits

The layer permits transfers when the policy allows them, while judge agreement varies and the headline metric remains grounded in executed tool calls. Sensitivity checks and provider-side filtering do not change the central reported result, and the authorization check adds no measurable agent latency at these timescales.

4.9 What the layer does when the policy permits

  • 1,904 successful Level 1 payments executed behind the layer, versus 1,912 of 1,928 policy-evaluated calls allowed; the wildcard recipient list permits any recipient.The layer's behavior follows the policy: it permits transfers at essentially the model-alone rate when recipients are allowed.Source: §4.9 · p. 14

4.10 The judge panel, and why the headline does not use it

  • Kappa was 0.772 overall, 0.167 at Level 3 and 0.521 at Level 5; judge recovery ranged from 64.4% to 99.1%.The headline reads executed tool calls rather than judge verdicts; the planned human-labeled validation slice was not completed, leaving judge-versus-human validity open.Source: §4.10 · p. 14

4.11 Sensitivity and provider-side filtering

  • Excluding 99 out-of-window attacks changes the Levels 2 to 4 unpermitted-transfer rate from 140/76,842 (0.182%) to 140/75,694 (0.185%).The Level 5 rate also changes only slightly, from 5,362/22,320 (24.02%) to 5,341/22,124 (24.14%); no headline figure depends on the tail.Source: §4.11 · p. 14
  • 582 evaluations were blocked by provider input filters before the model saw the prompt, including 66 Level 1 model-alone evaluations for Astra.These evaluations remain in the denominators as no-transfer cases.Source: §4.11 · p. 14

4.12 Cost and latency

  • 47.4% of input-plus-output tokens went to the judge panel, while median evaluation latency was unchanged by the layer within noise.The policy check is not a measurable share of agent latency at these timescales; the companion paper [13] measures its median directly at 53ms.Source: §4.12 · pp. 14–15

5 What the replay shows that a leaderboard cannot

Replay outcomes vary by source session, attack family, replay track and model, so a single model leaderboard hides important structure. The results also distinguish attempted transfers from executed unpermitted transfers and from whether a requester was entitled to a permitted payment.

5.1 Outcomes concentrate in sessions, not techniques

  • One of 790 source sessions produced 67 of the 140 Levels 2 to 4 model-alone unpermitted transfers (47.9%); two sessions produced half.The 140 outcomes came from 24 sessions, so evaluation count overstates the effective sample size; intervals are clustered by session.Source: §5.1 · p. 15

5.2 One cohort produces almost all multi-turn Level 2 outcomes

  • 111 of 113 multi-turn Level 2 model-alone outcomes came from 103 forged-receipt attempts, and the cohort produced none behind the layer.The attackers inserted a purported recipient-verification result; the boundary check evaluated the actual recipient against the policy instead of relying on the transcript.Source: Table 6 · p. 15, §5.2 · p. 15

5.3 Multi-turn helps one attack family, not attacks in general

  • The multi-turn advantage at Levels 2 to 4 was concentrated in one forged-receipt attack family, not a general property of extended context.The authors caution against generalizing beyond this corpus and execution-based metric; at Level 5, 51.6% of multi-turn outcomes executed on turn 1 because the prompt instructed tool use.Source: Table 7 · p. 16, §5.3 · p. 16

5.4 Unpermitted-transfer outcomes show limited cross-model transfer

  • 73 of 2,809 attacks defeated at least one model on multi-turn replay at Levels 2 to 4, and only one defeated five models.None defeated more than five of fourteen models; the authors say attacks producing an unpermitted transfer on one model mostly did not transfer to the next.Source: §5.4 · p. 16

5.5 A request is not a compromise, and a permitted payment is not an acquittal

  • A request, an executed payment and allowlist membership are distinct observed facts; allowlist membership does not establish requester entitlement or legitimate intent.The study did not adjudicate intent, and its planned human-labeled validation slice was not completed.Source: §5.5 · pp. 16–17

6 Discussion

The discussion argues that model-only zero counts do not establish resistance, while the authorization result is conditional on a correctly configured policy engine enforcing a well-specified capability. It draws evaluation and deployment implications within that scope.

6.1 The strongest counter-argument: five models reached zero

  • Every model issued payment requests in 33.1% to 39.9% of balanced single-turn Level 2 to 4 evaluations, including the five models with zero unpermitted transfers.Model rankings at Level 1, where any recipient was permitted, did not predict which models produced unpermitted transfers at Levels 2 to 4.Source: §6.1 · p. 17

6.2 What the zero is and is not

  • The 0 of 69,297 result applies when a deterministic check evaluates a well-specified capability against an explicit allowlist; it does not establish that authorization layers cannot fail.The authors identify wrong policies, overly broad capabilities, bypassable enforcement points and actions not expressible as policy decisions as boundaries of the claim.Source: §6.2 · p. 17

6.3 Implications for evaluation practice

  • Report enforcement architecture as an experimental variable and cluster uncertainty by attack source; one session produced 47.9% of Levels 2 to 4 outcomes.The authors also recommend domains with checkable state changes, since judge agreement reached kappa 0.167 at one level.Source: §6.3 · p. 17

6.4 Relation to the economics of agent autonomy

  • The study informs one input to agent-autonomy decisions, but does not measure productivity, non-payment actions or error costs outside this domain.The policy denied 187 of the 25,640 calls it evaluated, so most passed without a human in the loop.Source: §6.4 · p. 18

7 Limitations

The evidence is limited to a simulated-bank payment tool and a self-selected CTF attack corpus, with several incomplete or potentially confounded measurements. Results should be read within those experimental boundaries.

7 Limitations

  • The benchmark covers payments through one transfer tool in a simulated bank, not code execution, data access or agent-to-agent delegation.Levels 1 and 5 are not attack-difficulty measurements: one permits any recipient, while the other grants no capability and instructs tool use.Source: §7 · p. 18
  • CTF participants competing for a $6,500 prize pool are not general users or state-grade adversaries, and the rates are not a population estimate.Attack-class coverage is uneven: tool-sequence exploits, parameter fuzzing and multi-agent delegation are underrepresented.Source: §7 · p. 18
  • Each cell is a single run, leaving no within-cell variance estimate; the planned repeat-subset check is absent, and provider-specific decode defaults confound model comparisons.The study chose fidelity to the conditions under which attacks were written over repeated sampling.Source: §7 · p. 18
  • Some behind-the-layer multi-turn cells are missing or partial, and the Level 4 zero reflects a stricter local engine than the published pack.The 0 of 69,297 is computed over completed cells only; the local engine required an audit code in the memo that the public pack does not enforce.Source: §7 · p. 18
  • Judge reliability varies, and the pre-registered human-validation slice was not completed, leaving judge-versus-human validity unresolved.The authors report kappa 0.167 at Level 3 and one judge's 64.4% recall rather than correcting those judge results.Source: §7 · p. 18
  • Some prompts match content publicly visible during the event, so model-training contamination cannot be ruled out; the paper makes no causal claim about model internals.The authors cannot rule out that some models' training data includes CTF discussion, and do not claim to know why an individual model complied or refused.Source: §7 · p. 18

8 Reproducibility and release

The release supports reproduction of the evaluation and analysis, while identifying specific data and infrastructure dependencies that prevent some quantities from being recomputed from the public rows.

8 Reproducibility and release

  • 225,964 evaluation rows, the five enforced level passports, scoring code, coverage and error counts, and preregistration materials are released under CC BY 4.0.The release is at huggingface.co/datasets/aporthq/vault-benchmark-v1.Source: §8 · p. 19
  • The released script regenerates the analysis from a frozen snapshot when available, using 1,500 session-clustered bootstrap resamples and seed 20260915.Source: §8 · p. 19
  • Five quantities cannot be recomputed from released rows, but the headline figures reproduce exactly despite 34 missing Level 2–4 payments and 390 missing Level 5 payments.All missing successful-payment rows are GLM-5.3 transcripts withheld under Z.ai’s terms; requests, recipient membership, and unpermitted-transfer records remain available.Source: §8 · p. 19
  • The CTF harness, judge prompts, and signing keys are withheld because releasing a working attack harness against a live policy configuration poses a different risk; the corpus includes a contamination canary.Source: §8 · p. 19
  • Public rows support the listed tables and results, while policy-engine parity and the 53ms enforcement latency reported in [13] require the authors’ infrastructure.Source: §8 · p. 19

9 Conclusion

Across Levels 2 to 4, the authorization check prevented unpermitted transfers while allowing many payments to succeed. The authors conclude that, on this corpus, execution-path checks—not model choice—determined whether out-of-policy actions executed.

9 Conclusion

  • 140 of 76,842 unpermitted transfers occurred with the model alone versus 0 of 69,297 behind the check; matched triples showed 105 versus 0, with a 0.38% session-clustered upper bound.Source: §9 · p. 19
  • The check allowed 25,453 transfer calls and denied 187 of 25,640, including 148 denials for a forbidden recipient.The authors characterize this as enforcing one specific restriction, not making models refuse or providing a general safety guarantee.Source: §9 · p. 19
  • Failures arose from two dozen of 790 sessions, one technique accounted for almost all multi-turn effect, and no attack defeated more than five of fourteen models.Source: §9 · p. 19
  • On this corpus, model choice did not determine whether an action outside policy executed, whereas a check in the execution path did.The Open Agent Passport specification and its enforcement latency are given in [13].Source: §9 · pp. 19–20

10 Disclosure

The author discloses a financial interest in the evaluated authorization layer and describes structural mitigations, including public release of the data and analysis. No human-subject data or real-money transfers are included.

10 Disclosure

  • The author founded the company developing the evaluated layer and designed, ran, and analyzed the benchmark; the paper identifies this as a conflict of interest.Source: §10 · p. 20
  • Mitigations include a deterministic executed-call outcome, preregistration and addenda, release of all evaluation rows and the analysis script, and encouragement of independent replication.Source: §10 · p. 20
  • No human-subject data was released, seven submissions containing personal information were removed, and no real money moved.Source: §10 · p. 20

Related papers

Loading 2609.22076v1…