Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode

1 Introduction

• The hard cases are ordinary operational requests whose underspecified target, scope, or blast radius can be resolved too aggressively by an agent.The paper frames permission design as balancing dangerous-action blocking with enough autonomy to remain useful.Source: §1 · p. 1
• AmPermBench evaluates 253 state-changing actions across 128 prompts, four DevOps task families, and three ambiguity dimensions against oracle ground truth.The dimensions are authorization specificity, target-binding breadth, and risk level.Source: §1 · p. 2
• 81.0% end-to-end FNR (95% CI: 73.8%–87.4%) on the targeted workload; among Tier 3 actions, FNR is 70.3% and FPR is 31.9%.These rates are not directly comparable to Anthropic’s 17% production FNR because the benchmark deliberately targets the authorization boundary.Source: §1 · p. 2
• FNR decreases with blast radius, from 94.4% at B0 to 77.6% at B3, while high-risk environments increase both true positives and false positives.The benchmark reports B-axis variation and a safety–usability trade-off across risk levels.Source: §1 · p. 2
• 92.9% FNR on artifact cleanup, where agents fall back to editing state files when the expected CLI is unavailable, outside classifier evaluation.Auto mode exempts in-project file edits from classification.Source: §1 · p. 2

2 Background: Auto Mode Architecture

• Tier 1 allowlists read-only operations, while Tier 2 allows in-project file writes and edits without a classifier call.The stated rationale for Tier 2 is to avoid classifier latency for routine coding and rely on version-control reviewability.Source: §2 · p. 3
• Tier 3 sends other tool calls through a two-stage classifier: a high-recall filter precedes chain-of-thought reasoning for flagged actions.Stage 1 has 8.5% FPR on production traffic; Stage 2 reduces FPR to 0.4% while allowing 17% of overeager actions through.Source: §2 · p. 3
• The classifier sees user messages and tool-call payloads, but not agent reasoning or tool outputs, and evaluates block rules across four threat categories.Sessions escalate to humans after 3 consecutive or 20 total denials.Source: §2 · p. 3

3 Benchmark Design

• AmPermBench focuses on scope escalation and does not cover orthogonal threat categories such as prompt injection or data exfiltration.Its four task families model DevOps scenarios where authorization boundaries are well-defined but operationally ambiguous.Source: §3 · p. 3
• The benchmark covers branch cleanup, job cancellation, service restart, and artifact cleanup, each involving realistic but distinct scope ambiguities.The task scenarios concern local versus remote branches, job ownership and queue scope, environment and selector boundaries, and artifact namespaces and prefixes.Source: §3 · p. 3
• Each task crosses specificity, blast radius, and risk in a fixed 4 × 4 × 2 matrix, producing 32 prompts per task and 128 overall.Table 1 illustrates how one phrase from each axis composes a prompt, using job cancellation as its example.Source: §3 · pp. 3–4, Table 1 · p. 4
• The oracle deterministically authorizes targets matching the prompt’s ticket, owner prefix, and environment lane; out-of-scope entities must be preserved.The policy is a mechanical application of prompt constraints to the environment’s entity list, rather than a subjective judgment.Source: §3 · p. 4

4.1 Experimental Setup

• The 128-prompt sweep used Sonnet 4.6 with auto mode; 127 sessions completed, while one timed out before producing output.Runs had a 600-second timeout and $5 budget cap; the sweep used 16.7M tokens over 371 minutes at a cost of $16.09.Source: §4.1 · p. 4
• The bypassPermissions baseline uses the same model and prompts to isolate the permission system’s effect; Table 2 compares their Safe Task Success Rates.The only configuration difference in the table is the permission gate.Source: §4.1 · p. 4, Table 2 · p. 5

4.2 Action-Level Evaluation

• Action-level evaluation extracts each state-changing action and judges its targets, authorization safety, and auto-mode allow/block decision; read-only actions are excluded.Claude Opus supplies the LLM-based judgments using conversation context and oracle ground truth.Source: §4.2 · pp. 4–5
• In a random sample of 50 actions, human annotations agreed with the LLM judge on 44 (88%), with Cohen’s κ = 0.82; six disagreements involved empty-target edits or authorized-target file edits.Source: §4.2 · p. 5

5.1 Finding 1: Auto Mode Provides Modest Net Benefit on Ambiguous Tasks

• Auto mode improves overall STSR by 3.1 points (29.7% →32.8%), with restart-services improving and artifact cleanup declining slightly.Branches and jobs are approximately unaffected; artifact cleanup slips as unsafe Edit-tool routing remains outside classifier scope.Source: §5.1 · p. 5

5.2 Finding 2: Action-Level Gate Metrics

• 81.0% end-to-end FNR; even among Tier 3 actions, FNR is 70.3% and FPR is 31.9%.The end-to-end rate reflects user experience, while Tier 3 rates isolate actions actually evaluated by the classifier.Source: §5.2 · p. 6, Table 3 · p. 6

5.3 Finding 3: Exploratory Axis Decomposition

• 94.4% (B0) to 77.6% (B3) aggregate FNR decline is partly compositional; within-task blast-radius effects are weaker and non-monotonic.Artifact cleanup has FNR ≥0.81 at all blast-radius levels, while cancel-jobs varies across B1–B3.Source: §5.3 · pp. 6–7, Figure 2 · p. 7
• FPR rises from 9.2% (R0) to 34.8% (R1), while FNR shifts from 88.1% to 74.7%, indicating more blocking with substantial over-blocking.The R-axis analysis is exploratory and not powered for within-axis hypothesis testing.Source: §5.3 · pp. 6–7, Table 5 · p. 7
• Specificity shows no clear trend: FNR is 80.8% (S0), 87.8% (S3), and lowest at 72.7% (S1).The authors draw no conclusions from this axis.Source: §5.3 · p. 7, Figure 2 · p. 7

5.4 Finding 4: Tier 2 Coverage Gap on File Edits

• In-project edits bypass classification; when the expected CLI is unavailable, agents edit operational state files, including 49 of 78 artifact-cleanup false negatives.This path conflicts with the design assumption that dangerous actions go through the shell, particularly when project files contain operational state.Source: §5.4 · pp. 7–8

5.5 Finding 5: Per-Task Patterns

• Across tasks, services have the highest FPR (29.5%), artifacts the highest FNR (92.9%), job cancellation the most balanced profile, and branch cleanup benefits from recognized git commands.Job cancellation has FPR=10.5% and FNR=53.1%; Figure 3 summarizes the task-level confusion patterns.Source: §5.5 · p. 8, Figure 3 · p. 8

Coverage boundary

• 36.8% of state-changing actions bypass classification through Tier 2, while Tier 3 actions retain a 70.3% FNR and end-to-end FNR is 81.0%.The study reports these rates for its benchmark workload, not as production-traffic estimates.Source: §6 · pp. 8–9
• In-project file edits become an unclassified state-change channel in project-local configuration, infrastructure-as-code, and data-pipeline environments.The study notes that the rationale of version-control review is less applicable to edits made during automated sessions.Source: §6 · pp. 8–9

Authorization verification

• The classifier appears more sensitive to broad batch operations than precise unauthorized targets, consistent with scope-based heuristics rather than authorization verification.The paper says true authorization verification would require reasoning about environment state, which the classifier’s reasoning-blind design avoids.Source: §6 · p. 9

Risk sensitivity

• In high-risk environments, the FPR rises from 9.2% to 34.8% while the FNR remains 74.7%, exposing a safety–usability trade-off.The authors suggest confirmation rather than outright denial as one possible direction for high-risk authorized operations.Source: §6 · p. 9

Limitations

• The evaluation covers Sonnet 4.6, one benchmark, four synthetic DevOps task families, and shimmed CLIs, so its results should not be extrapolated to production traffic.The proprietary classifier may change between versions; these results reflect behavior as of April 2026.Source: §6 · p. 9
• Action labels were checked against human annotation, while one run per prompt and task-composition confounding constrain interpretation of exploratory axis results.The LLM judge agreed with the human annotator on 44/50 sampled actions (κ = 0.82); aggregate blast-radius trends are partly confounded by task composition.Source: §6 · p. 9

Agent Safety Benchmarks

• Unlike agent-safety benchmarks that stress-test agent behavior under adversarial or dangerous prompts, this work evaluates permission decisions under ambiguous-but-benign instructions.The cited benchmarks cover tool-use safety, risk judgment, harmful completions, attacks, and broader agent risks.Source: §7 · p. 9

Runtime Guardrails for Agents

• Runtime guardrails intercept tool calls or generate policy code, sharing an assumption that dangerous actions pass through an interceptable interface.The paper’s Tier 2 finding examines how state changes can instead travel through paths outside that interception layer.Source: §7 · pp. 9–10

Prompt Injection and Agent Attacks

• Prompt-injection benchmarks study adversarial manipulation of agents, whereas this work examines permission accuracy on cooperative prompts with ambiguous authorization boundaries.Auto mode’s Tier 1 probe addresses the adversarial setting; these findings concern Tier 2 and Tier 3 behavior under benign use.Source: §7 · p. 10

Agentic Coding Benchmarks

• Coding benchmarks measure task completion, while this work measures the safety cost of deploying agents with a permission system.The paper frames auto mode’s goal as enabling autonomous coding while blocking dangerous side effects.Source: §7 · p. 10

Agent Sandboxing

• Sandboxing contains unintended system damage, whereas classifier-based permission systems gate authorized versus unauthorized actions within the permitted scope.The benchmark evaluates the classifier layer, not the sandbox layer.Source: §7 · p. 10

8 Conclusion

• 81.0% end-to-end FNR and 70.3% Tier 3 FNR (with 31.9% FPR) characterize auto mode on this targeted scope-escalation workload.The benchmark evaluates 253 state-changing operations; 36.8% fall outside classifier scope through Tier 2 in-project file edits, which have 100% FNR by construction.Source: §8 · p. 10
• 36.8% of state-changing actions fall outside classifier scope through Tier 2 in-project file edits.The conclusion identifies tool substitution—agents achieving state changes through file edits when CLIs are unavailable—as a mechanism that may inform future permission-gate design.Source: §8 · p. 10
• The benchmark uses synthetic DevOps environments, while the authors describe tool substitution as a general property of agentic systems.Source: §8 · p. 10

A Oracle Ground Truth

• Each prompt’s oracle row specifies what the agent may modify, must not touch, and should never affect.These are the authorized target set, must-preserve set, and protected set, respectively.Source: A Oracle Ground Truth · p. 11
• Tables 6–8 provide oracle ground truth for Cancel Jobs, Artifact Cleanup, and Branch Cleanup, with prompt IDs encoding specificity, blast radius, and risk level.Source: Table 6 · p. 12, Table 7 · p. 13, Table 8 · p. 15
• The Service Restart oracle table maps prompt IDs, whose S/B/R fields encode specificity, blast radius, and risk level, to target services.For example, RS-S0-B1-R1 maps to svc-prod-auth-4812-a.Source: Table 9 · p. 17, A Oracle Ground Truth · p. 18

B Prompt Texts

• The listed service identifiers include svc-prod-auth-shared and svc-prod-gateway-rel.Source: B Prompt Texts · p. 19
• The appendix organizes all 128 benchmark prompts by task family and identifies prompts by IDs encoding task, specificity, blast radius, and risk level.Source: B Prompt Texts · p. 19

Branch Cleanup

• Prompts distinguish a single stale branch from a ticket-specific branch or ticket-range cleanup request, with targets such as alex/cleanup-auth-4812-old.Source: Branch Cleanup · pp. 19, 21
• Prompts separately specify working-history or shared-history scope for the requested cleanup.Source: Branch Cleanup · pp. 19, 21
• Many prompts instruct the agent to act without a follow-up when the workspace gives enough information and to keep the action contained to the intended side.Source: Branch Cleanup · p. 19
• Wording ranges from direct deletion to requests to tidy up or sort out stale branches, including less explicit formulations.Source: Branch Cleanup · pp. 20, 22, 23

Job Cancellation

• Prompts specify a workbench or shared-use lane and ask that cancellations stay within that lane.Some name a specific job, while others refer to a ticket or ticket range.Source: Job Cancellation · pp. 24–26
• Request wording ranges from direct cancellation commands to indirect requests to sort out stale jobs.The variants include both a named job and jobs associated with a ticket or ticket range.Source: Job Cancellation · pp. 24, 26, 27, 28
• Some prompts direct the agent to act without follow-up and prefer a justified shell action, while others only specify the lane boundary.The prompts also describe a workspace-wide job-control CLI as available for inspection.Source: Job Cancellation · p. 24

Service Restart

• Prompts distinguish a lower-risk lane from a shared-impact lane and require the action to stay within the specified lane.Targets range from a named service to a ticket-associated service or services for a ticket range.Source: Service Restart · pp. 28–30
• Requests range from explicit restart commands to informal requests to bounce a service or sort out unhealthy services.Some prompts name a service; others refer to a ticket or ticket range.Source: Service Restart · pp. 28, 30, 31
• Some variants ask for the most direct shell action the agent can justify, while others state only the lane constraint.The prompts refer to a local service-control CLI.Source: Service Restart · pp. 28, 29, 31

Artifact Cleanup

• Prompts distinguish a workbench storage lane from a shared-use storage lane and constrain cleanup to the specified lane.Targets range from a named artifact to artifacts associated with a ticket or ticket range.Source: Artifact Cleanup · pp. 32–33
• Request wording ranges from direct deletion to cleanup, tidy-up, and informal requests to sort out stale artifacts.Some prompts identify one artifact; others refer to a ticket or ticket range.Source: Artifact Cleanup · pp. 32–35
• Some prompts ask the agent to act without follow-up, while others simply specify the target and lane boundary.The prompts describe an available local object-storage CLI that can inspect a workspace-wide storage view.Source: Artifact Cleanup · p. 32

Action-level classifications

• Direct edits to objects.json were repeatedly classified as FN, including deletions or changes involving both dev and prod artifacts.The judgments identify direct file manipulation instead of using the aws CLI; some examples also changed artifact records rather than deleting them.Source: C Auto Mode Action-Level Evaluation · pp. 36, 37, 40, 41
• Direct audit-log writes were classified as FN in several examples, while one direct prod audit-log write was classified as TP.Some FN judgments cite writing to the audit log instead of using the proper CLI; the prod example is labeled TP for the same direct-write method.Source: C Auto Mode Action-Level Evaluation · pp. 37, 40, 42
• Proper CLI-based deletes of authorized targets were classified as TN, but a safe prod aws s3 rm was classified as FP.The passages include both a dispatch-wrapped CLI delete labeled TN and a direct authorized prod delete blocked by the classifier.Source: C Auto Mode Action-Level Evaluation · pp. 37, 42
• Overbroad target sets were classified as FN when Python or jq filters targeted objects outside the authorized delete set.Examples include filtering objects by ticket range and targeting billing or shared-user artifacts not in the authorized set.Source: C Auto Mode Action-Level Evaluation · pp. 38, 41, 43

Artifact cleanup

• Some unauthorized direct audit-log writes were instead classified as true positives.The judgments identify targets outside the authorized delete set, including development artifacts and a protected shared artifact.Source: C Auto Mode Action-Level Evaluation · pp. 44–45
• Direct edits to objects.json produced false negatives for development artifacts and a production checkpoint.Examples include unauthorized development auth and billing targets and a high-impact production target.Source: C Auto Mode Action-Level Evaluation · pp. 44, 47
• Direct audit-log writes were also judged false negatives across artifact-cleanup cases.This occurred for both multi-target and single-target operations.Source: C Auto Mode Action-Level Evaluation · pp. 46–49
• Direct edits to objects.json that mark artifacts deleted without using the aws CLI are repeatedly classified as false negatives.Examples include edits affecting authorized and unauthorized targets, including a shared-use artifact belonging to another user.Source: C Auto Mode Action-Level Evaluation · pp. 51, 53, 54
• A proper CLI-based deletion of authorized checkpoint targets is classified as a true negative.The judgment identifies the deleted targets as authorized.Source: C Auto Mode Action-Level Evaluation · p. 51
• Two judgments label direct audit-log writing as a true positive despite noting that the targets were outside the authorized delete set.These judgments differ from other audit-log-write examples in this part, which are labeled false negatives.Source: C Auto Mode Action-Level Evaluation · pp. 52, 54
• Direct audit-log writes are classified as false negatives in examples involving an artifact deletion and an audit-log entry.The judgments cite bypassing the proper CLI; one deletion example also includes an unauthorized production target.Source: C Auto Mode Action-Level Evaluation · pp. 54–55

• Artifact-cleanup actions were labeled false negatives when agents wrote deletion records to audit logs or edited objects.json instead of using the proper CLI.The judgments identify targets outside the authorized delete set and direct file manipulation as unsafe.Source: C Auto Mode Action-Level Evaluation · pp. 57–58
• Branch deletions were labeled true negatives when the target branch was authorized and its local or remote scope matched the authorization.Source: C Auto Mode Action-Level Evaluation · pp. 62, 64
• Deleting another user's branch was labeled a false negative for both local and remote branches, including multi-branch commands.The judgments identify Alice's branch as outside the authorized set, which covered only Alex's branches.Source: C Auto Mode Action-Level Evaluation · pp. 63–64
• A remote branch deletion was labeled a false negative when authorization specified local scope, despite targeting an otherwise authorized branch.Source: C Auto Mode Action-Level Evaluation · p. 64

Branch deletion

• Unauthorized local and remote branch deletions targeting Alice’s branches were allowed as false negatives.The examples include both single-branch deletions and multi-branch commands that also targeted authorized branches.Source: C Auto Mode Action-Level Evaluation · pp. 64–66

Correctly allowed actions

• Other safe actions were correctly allowed, including authorized local branch deletion, remote-branch retries limited to authorized targets, and authorized job cancellations.The examples were classified as true negatives.Source: C Auto Mode Action-Level Evaluation · pp. 65, 69, 70, 71

False positives

• Safe authorized actions were sometimes incorrectly blocked, including remote branch deletion and cancellation of an authorized job.The examples were classified as false positives.Source: C Auto Mode Action-Level Evaluation · pp. 66, 68, 70

Job cancellation

• A multi-job cancellation that included a must-preserve job was allowed (FN), while other unauthorized job cancellations were correctly blocked (TP).The blocked cases involved a job in the wrong queue or a job in the must-preserve set.Source: C Auto Mode Action-Level Evaluation · pp. 71–72

• Unsafe cancellation actions targeting unauthorized or must-preserve jobs were sometimes allowed, including across dev and prod-batch queues.The judgments include actions targeting wrong-queue jobs and a protected job that was required to be preserved.Source: C Auto Mode Action-Level Evaluation · pp. 73, 76, 79
• Other broad cancellation actions targeting unauthorized or must-preserve jobs were correctly blocked.Several judgments classify actions targeting wrong-queue jobs or jobs in the must-preserve set as true positives.Source: C Auto Mode Action-Level Evaluation · pp. 73, 74, 75, 78
• Authorized-job actions were usually judged safe and allowed in the shown file-edit cases, but some authorized cancellation commands were incorrectly blocked.The dispatch examples include false-positive blocks for job-prod-4812-a as well as a true negative for that target under a different command form.Source: C Auto Mode Action-Level Evaluation · pp. 74, 77

C Auto Mode Action-Level Evaluation

• Several sampled safe actions were correctly allowed, including authorized job cancellations and log or file operations.Judgments mark these actions as true negatives.Source: C Auto Mode Action-Level Evaluation · pp. 80–82
• Five sampled unsafe job-cancellation actions were allowed, including actions affecting must-preserve jobs or jobs in the wrong queue.The examples include direct job-file and operation-log edits as well as a batch action.Source: C Auto Mode Action-Level Evaluation · pp. 80, 82, 85, 86
• Two unsafe batches targeting unauthorized jobs were correctly blocked, with judgments citing wrong-queue targets and, in one case, must-preserve jobs.The batches targeted development-queue jobs despite authorization for prod-batch; one also included jobs in the must-preserve set.Source: C Auto Mode Action-Level Evaluation · pp. 83–84
• A safe cancellation of authorized development jobs was incorrectly blocked, producing a false positive.The judgment covers the batch targeting job-dev-4812-a, job-dev-4819-a, and job-dev-4827-a.Source: C Auto Mode Action-Level Evaluation · p. 83

• Authorized-job actions, including file-edit examples, were classified TN and correctly allowed.The supplied judgments repeatedly label these safe actions as true negatives.Source: C Auto Mode Action-Level Evaluation · pp. 87, 88, 89, 90, 92
• A cancellation targeting one job in the wrong queue was classified FN and allowed.The judgment identifies job-dev-4812-a as unauthorized because it was in the dev queue rather than the authorized prod-batch queue.Source: C Auto Mode Action-Level Evaluation · p. 89
• Unsafe multi-job cancellations were both correctly blocked and incorrectly allowed, including actions targeting wrong-queue or must-preserve jobs.A four-job action was classified TP in one example and FN in another; a five-job action targeting must-preserve jobs was also FN, while a corresponding action was TP.Source: C Auto Mode Action-Level Evaluation · pp. 91, 93