PaperlayerView exact revision on arXiv

Source-linked AI summary

Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments

Dotan Davidovich, Yair Amar, Hai Rozencwajg, Or Hiltch

arXiv:2608.02670v1cs.CRcs.AI

TL;DR

Existing evaluations leave uncertain how coding-agent performance changes when security policies are enforced. This paper introduces Boundary-Bench to evaluate agents across nested policy levels, finding that hardening worsens success and cost non-uniformly and changes the trade-off across bundles.

  • Problem

    Existing benchmarks do not show how coding-agent performance changes as security policies tighten, despite agents increasingly operating under organizational controls.

    Source: Introduction · p. 1

  • Method

    Boundary-Bench evaluates native model–harness bundles under nested, operating-system-enforced policies and audits strict-policy task solvability.

    Source: Introduction · p. 2, Boundary-Bench · p. 2, Benchmark Audit · p. 3

  • Results

    Every bundle loses success and gains cost under NIST-derived high, with success losses reaching 18.3 points and cost inflation 167.3%; failures mainly grind into timeouts or wrong solutions.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    Model selection depends on the deployment policy, and reference-solution compatibility can identify affected tasks before agent runs for benchmarks that ship reference solutions.

    Source: Conclusion · p. 7

    Evidence comes only from Terminal-Bench and its 89 tasks, so the findings are established for this benchmark only.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Coding agents increasingly run inside organizations whose security controls (scoped credentials, restricted egress, read-only filesystems, non-root execution) constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is unknown how performance changes when policy is enforced. In this work, we evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions. Hardening is never free but far from uniform: under the strictest policy, success losses reach 18.3 points and cost inflation 167.3\%, and the two axes disagree; the model that best preserves success is also the one that loses the most efficiency, so model choice is policy-dependent. Beyond aggregate scores, we characterize how agents behave when policy blocks their actions and decompose the failures hardening induces: runs grind into timeouts or wrong solutions rather than stopping early, in a mix that differs by model. To ground comparisons, we verify task solvability under the strictest policy, separating model failures from tasks the policy forecloses. We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents on Terminal-Bench and compatible benchmarks.

Introduction

Existing evaluations do not show how coding-agent performance changes as security policies tighten, even though agents increasingly operate under controls that restrict access and execution. Boundary-Bench evaluates agents across nested policy levels and tracks both performance and task-level diagnostics.

Introduction

  • Organizations’ security controls—including scoped credentials, network restrictions, protected paths, and limited privileges—constrain coding agents as they do other software.The paper calls environments where policy restricts what agents can observe, access, or execute “hardened environments.”Source: Introduction · p. 1
  • Existing benchmarks leave security configuration fixed or permissive, so they do not show how coding-agent performance changes as policies tighten.Compliance-scored benchmarks also neither enforce policies at runtime nor measure their operational cost.Source: Introduction · p. 1
  • Hardening worsens every model–harness bundle’s success–cost operating point, but by differing amounts and in different currencies, making model choice policy-dependent.Agents respond differently to denied actions, so unconstrained scores do not predict performance under enforced policy.Source: Introduction · p. 1
  • Boundary-Bench evaluates native model harnesses under three nested policy levels and reports task outcomes, accessibility, reference-solution compatibility, and policy-induced failure modes.The evaluation runs on Terminal-Bench 2.1, with policies enforced by native Linux mechanisms.Source: Introduction · p. 2

Related Work

Prior evaluations study coding performance, safety, policy compliance, robustness, and enforcement, but generally hold execution security fixed or assess a single configuration. This paper varies policy severity across bundles while measuring success, cost, and task accessibility.

Related Work

  • Coding benchmarks and cost-aware evaluations hold execution-security configuration fixed rather than varying it as an evaluation factor.The paper also audits and repairs five over-specified task verifiers before policy comparisons.Source: Related Work · p. 2
  • Safety and policy evaluations differ from this work: they assess harmfulness, risky tool use, or policy adherence rather than success and token spend under enforced graded policies.The paper imposes compliance as a graded property of the execution environment.Source: Related Work · p. 2
  • Robustness studies vary inputs or inject synthetic faults, whereas this work varies the security policy itself and holds it fixed during each evaluation run.The paper draws qualitative inspiration from the effective-robustness lens.Source: Related Work · p. 2
  • The paper frames policy severity as a shared, nested evaluation axis, complementing prior enforcement studies that price one configuration in isolation.The comparison measures enforcement costs across model–harness bundles and tracks task accessibility at each level.Source: Related Work · p. 2

Boundary-Bench

Boundary-Bench layers operating-system-enforced policies onto an existing coding-agent benchmark and measures changes in success and cost against an unrestricted baseline. Its graded policies restrict network, filesystem, and privilege capabilities, and its strict-policy audit distinguishes task infeasibility from agent failure.

Boundary-Bench

  • Boundary-Bench layers configurable, operating-system-enforced policies onto coding benchmarks to measure changes in agent success and cost relative to an unrestricted baseline.The framework is designed to attach to an existing benchmark.Source: Boundary-Bench · p. 2

Policy Enforcement

  • The policy ladder progresses from an open control, to non-root execution, to NIST-derived high restrictions on egress, filesystem access, and privileges.The three modeled axes are network egress, filesystem scope, and privilege; the strictest step changes multiple surfaces together.Source: Policy Enforcement · pp. 2–3

Benchmark Audit

  • Before interpreting strict-policy results, the audit tests reference-solution solvability, checks apparent verifier false failures, and classifies failures by termination time.Tasks without a policy-compliant solution are blocked by design; failures reaching 95% of the wall-clock budget count as timeouts.Source: Benchmark Audit · p. 3

Experimental Setup

The experiments compare twelve frozen model–harness bundles across 89 Terminal-Bench tasks, using three valid trials per bundle, task, and policy. Robustness is measured as within-bundle changes in success and cost, with task-level cost expressed as a multiplier.

Experimental Setup

  • Twelve frozen model–harness bundles run on all 89 tasks under every policy, with three valid trials per bundle–task–policy cell.The primary comparison pairs models with their native harnesses and uses high effort.Source: Experimental Setup · p. 3
  • A trial continues until the harness returns or the task timeout is reached, and the official verifier scores the resulting environment; invalid malfunctions are excluded and rerun.A timed-out execution can still pass, and harness-internal retries remain part of the trial.Source: Experimental Setup · p. 3
  • Cost-distribution sampling uses 100 runs per condition on two tasks with GPT-5.6 Luna–Codex, while the harness ablation crosses Luna and Grok 4.5 with Codex and Grok Build.The sampled tasks differ in how sharply success degrades under hardening; the ablation tests model versus harness effects at two cost extremes.Source: Experimental Setup · p. 3

Robustness Measure

  • Each bundle’s success rate and mean cost per task are compared with that bundle’s own control values; success shifts are percentage-point differences and cost shifts are normalized by control cost.Task-level cost is the policy-to-control mean-cost multiplier for completed runs, averaged equally across bundle–task pairs.Source: Robustness Measure · p. 3
  • Figure 1 plots success rate against total cost on a logarithmic scale, with marker shape encoding policy and color encoding model.Each point represents a bundle-policy combination, and dashed lines trace each policy’s Pareto frontier.Source: Figure 1 · p. 4

Results

Under policy, all evaluated bundles shift toward lower success and higher cost, but the trade-off varies by model. The results also distinguish policy-foreclosed tasks from solvable ones and trace added failures, costs, and blocked actions.

The Success–Cost Frontier under Policy

  • The success–cost frontier persists under every policy and shifts lower and further right as restrictions tighten, with the best operating point depending on the bundle.Grok 4.5 joins the high-cost end of the frontier under both hardened policies; on the 82 witnessed tasks it ties GPT-5.6 Sol under NIST-derived high and is costlier.Source: The Success–Cost Frontier under Policy · p. 4

Restriction Sensitivity

  • Under NIST-derived high, all twelve bundles lower success and raise mean cost per task; Grok 4.5 loses least (−7.1 pp) but has largest cost inflation (+167.3%), while Claude Sonnet 5 loses most (−18.3 pp) at +21.4%.GPT-5.6 Luna has the smallest cost inflation (+16.0%) with a success loss nearly as large as Grok 4.5’s; the axes do not move together across bundles.Source: Restriction Sensitivity · pp. 4–5

Benchmark Artifacts under Policy

  • Under NIST-derived high, 82 of 89 tasks have a solvability witness, while seven are blocked by design because no policy-compliant solution exists.The 50 unchanged canonical solutions and 32 author-written alternatives define unaffected and affected subsets, respectively.Source: Benchmark Artifacts under Policy · p. 5
  • On the 82 witnessed tasks, excluding the seven blocked tasks leaves a control-to-hardened success drop of 2.4 points for Grok 4.5 to 13.5 for Claude Sonnet 5.Verifier repairs raise success on two of five repaired tasks under NIST-derived high; the other three remain unchanged, including genuine hardening failures.Source: Benchmark Artifacts under Policy · p. 5

Mechanisms of Degradation

  • On witnessed tasks, hardening reduces success by 7.4 points (72.5% to 65.1%); additional failures are mostly timeouts (59%) or wrong solutions (37%), not early stops.Among matched passing runs, hardening adds 13% wall-clock time, 14% tool calls, and 26% tokens; the failure mix differs by bundle, with Terminus-2 bundles adding most timeouts and Codex bundles mostly wrong solutions.Source: Mechanisms of Degradation · p. 6

Cost Inflation under Policy

  • On two sampled tasks, median per-run cost rises 2.9× and 3.0× under NIST-derived high, and passing and failing runs pay similar increases.Workaround construction accounts for 97% of the added cost, as agents rebuild toolchains or substitute data after blocked paths.Source: Figure 3 · p. 6, Cost Inflation under Policy · p. 6
  • Cost and success penalties concentrate on affected tasks: their success falls from 70.3% to 54.5%, while the unaffected subset changes from 73.8% to 71.8%.The affected subset’s mean cost multiplier reaches 2.59× under NIST-derived high, versus 1.14× for unaffected tasks; 83.5% of additional failures occur on affected tasks.Source: Cost Inflation under Policy · pp. 6–7, Figure 4 · p. 7

Blocked Actions under Policy

  • Under NIST-derived high, 675 of 2,051 trials record a blocked action; egress denials lead, and their repeated occurrence accounts for most denial volume.Blocked-action exposure rises from 14.4% on unaffected tasks to 51.6% on affected tasks and 67.8% on tasks blocked by design; these are diagnostic associations, not causes.Source: Blocked Actions under Policy · p. 7

Limitations

The evaluation is limited to Terminal-Bench, frozen model–harness bundles, and one operating configuration. Its policy ladder measures performance cost, not security effectiveness.

Limitations

  • Findings are established only for Terminal-Bench’s 89 tasks, although Boundary-Bench attaches to runtime rather than task content.Adding benchmarks would require repeating the solvability audit, witness authoring, and inference spend.Source: Limitations · p. 7
  • Full-roster effects remain attributable to frozen model–harness bundles, not to model weights or the harness alone.Changing a harness also changes its prompt, tools, and interface; the ablation covers only the two cost extremes.Source: Limitations · p. 7
  • The primary comparison uses high reasoning effort and three valid trials per bundle–task–policy cell, with no other effort or trial count evaluated.Pareto, sensitivity, and ordering statements are descriptive, and small bundle differences should not be over-read.Source: Limitations · p. 7
  • The three policy conditions are a severity ladder derived from common enterprise restrictions, not a certified deployment configuration.The study measures performance cost, not security effectiveness.Source: Limitations · p. 7

Conclusion

Across twelve model–harness bundles on Terminal-Bench 2.1, strict policy worsens success and cost, but the trade-off varies by bundle. The paper links these outcomes to task compatibility and describes the policy’s native enforcement and verification.

Conclusion

  • Success losses reach 18.3 points and cost inflation reaches 167.3% under NIST-derived high, with every bundle losing success and gaining cost.The size of the changes differs across bundles.Source: Conclusion · p. 7
  • Hardening failures are longer runs ending in timeouts or wrong solutions, rather than early stops, so budgets must account for deployment policy.Some bundles absorb the trade-off as inflated cost, others as lost success.Source: Conclusion · p. 7
  • Reference-solution compatibility identifies affected tasks before agent runs for both success and cost, using a procedure applicable to benchmarks with reference solutions.The conclusion frames this as a predictable policy tax.Source: Conclusion · p. 7
  • The authors argue that benchmarks should include a policy axis because hardening restores discriminative headroom and model selection depends on deployment policy.They recommend reporting performance in a common restricted environment.Source: Conclusion · p. 7

Appendix A: Policy Construction Details

  • The appendix describes NIST-derived high through its network, filesystem, and privilege hardening axes, their native enforcement, and pre-flight probes.The probes verify the restrictions before the agent starts.Source: Appendix A · p. 10

Quality Control

  • Pre-flight probes test denials, bypass attempts, and permitted operations before an agent starts; a failed required probe discards the attempt as an infrastructure error.A hardening step that cannot be established fails closed rather than running unhardened.Source: Quality Control · p. 10

Axis Definitions

  • The high-policy overview organizes runtime restrictions across network, filesystem, and privilege axes and maps them to NIST controls and ordinary system errors.The agent observes errors such as HTTP 403, EROFS, and EPERM.Source: Axis Definitions · p. 10, Table 4 · p. 11

Appendix B: Solvability Witnesses under NIST-derived high

The solvability audit separates failures caused by model behavior from tasks foreclosed by policy or verifier assumptions. Its examples show how hardening can remove a required destination, package source, or task-presumed architecture.

Appendix B: Solvability Witnesses under NIST-derived high

  • The audit replays each shipped reference under NIST-derived high and authors a policy-compliant alternative when needed, counting either trajectory if it reaches the official passing state.Adapted witnesses use permitted operations without changing the task environment or verifier.Source: Appendix B · p. 14
  • Residual failures have two causes: task-policy conflicts where instructions require denied resources, and verifier over-specification where legitimate non-root solutions are not checked.The latter is illustrated by sqlite-with-gcov, whose verifier checks only the conventional root-owned location.Source: Appendix B · p. 14

Why solvable tasks still fail under policy

  • Five verifiers receive additive repairs that generalize root-owned path or working-directory resolution while leaving substantive assertions unchanged and applying the same repair across policy conditions.The original-verifier outcomes are preserved for audit.Source: Table 6 · p. 14, Why solvable tasks still fail under… · p. 15

Appendix B: Solvability Witnesses under NIST-derived high

  • 82 of 89 tasks admit a NIST-derived high solvability witness: 50 use the unmodified reference and 32 use authored solutions, leaving six task-policy conflicts and one scoring artifact.The remaining task, count-dataset-tokens, is separately excluded from the witness pool because its scoring artifact is documented in Appendix C.Source: Appendix B · p. 15

Why solvable tasks still fail under policy

  • Hardening removes different task assumptions: a writable destination for sqlite-with-gcov, package supply for mcmc-sampling-stan, and privileged architecture for configure-git-webserver.For mcmc-sampling-stan, CRAN’s primary mirrors returned HTTP 403 in 31 of 36 trials; the witness rebuilt configure-git-webserver’s stack inside the workspace.Source: Why solvable tasks still fail under… · p. 15
  • On sqlite-with-gcov, repair recovers 80.6% under the adapted verifier against 97.2% in control; for mcmc-sampling-stan, posterior checks mostly pass but every fresh-R-session check fails.The agents installed packages in separate directories that fresh R processes could not discover; a wrapper setting the library path passes the witness.Source: Why solvable tasks still fail under… · p. 15

Appendix C: A Benchmark-Integrity Artifact

The paper flags a benchmark-integrity artifact when the official scoring path awards success without establishing that the requested work occurred. The count-dataset-tokens verifier accepts an expected constant even when the trial contains no successful retrieval or tokenization.

  • A benchmark-integrity artifact is a case where the official scoring path does not establish that the requested work was performed.Flagged tasks remain in every aggregate, with raw trials preserved for audit.Source: Appendix C · p. 19
  • 13 count-dataset-tokens trials pass by submitting the expected constant, 79586, despite a failed dataset request and no successful retrieval or tokenization.The verifier checks only whether that string appears in the answer file.Source: Appendix C · p. 19
  • The task remains in reported aggregates as an integrity artifact but is excluded from the 82-task witness pool separately on solvability grounds.Source: Appendix C · p. 19

Appendix D: Provider-Side Safety Interventions

Provider-side refusals and fallback routing affect the deployed model route and are retained in reported aggregates. Their incidence does not rise with policy severity, and retries, rerouting, and trial-ending refusals behave differently.

  • Provider interventions are properties of the deployed route rather than runtime policy, so affected trials remain in every aggregate and are annotated per trial.Source: Appendix D · p. 19
  • Of 43 flagged GPT-5.6 Sol trials, 25 streams resumed and completed after retry, and 23 of those reached a passing state.When retries are exhausted, the trial may abort mid-trajectory or end before a tool call.Source: Appendix D · p. 19
  • Claude Fable 5 fallback served 29.9% of requests, while 27.6% of its trials included at least one fallback-served completion.The fallback route is declared per request and results describe the gated product route.Source: Appendix D · p. 19
  • Fallback and refusal are distinct: fallback retries a declined request, while a refusal is a completed response that routing cannot retry; Opus 4.8 still refused protein-assembly.That task was refused by every Claude model evaluated.Source: Appendix D · pp. 19–20
  • Flagged trials across the five affected bundles number 28 under control, 41 under non-root, and 29 under NIST-derived high; the paper reports no increase with hardening.Four bundles are flat across conditions; counts are small and clustered by bundle and task, so they are described rather than tested.Source: Appendix D · p. 20, Table 9 · p. 20
  • All 98 interventions fall on 16 tasks, and 70 of 73 trial-ending interventions were scored failures; only 51 fell on tasks with a solvability witness.The other 19 failures were on tasks blocked by design and could not have passed under the policy.Source: Appendix D · p. 20

Appendix E: Software Versions

The evaluation freezes software components and sandbox configurations across models, policies, and repetitions, with a documented Claude Code version exception for Opus 5. Provider identifiers and vendor-family harness assignments specify the evaluated bundles.

Appendix E: Software Versions

  • Software components were frozen before runs and held fixed across models, policies, and repetitions, with pinned dependency closures and task-specific sandbox images.Each trial uses a fresh sandbox provisioned from the task’s Terminal-Bench 2.1 image and declared resources.Source: Appendix E · pp. 20–21

Claude Code version for Claude Opus 5

  • Claude Opus 5 alone used Claude Code 2.1.220 because version 2.1.206 did not recognize its model identifier; affected pre-response client failures were excluded as harness incompatibilities.Other Claude models remained on version 2.1.206.Source: Claude Code version for Claude Opus 5 · p. 21

Provider model identifiers

  • The primary comparison fixes harness assignment by vendor family: Claude Code for Anthropic, Codex for OpenAI, Grok Build for Grok 4.5, and Terminus-2 for open-weight models.Table 11 lists the exact provider identifiers passed to the OpenRouter API for all twelve models.Source: Provider model identifiers · p. 21

Appendix F: Full Per-Bundle Results under Policy

Appendix F reports per-bundle success and cost under each policy, then identifies the non-dominated bundles under NIST-derived high on the witnessed task pool.

Appendix F: Full Per-Bundle Results under Policy

  • Table 12 reports per-bundle success rate and mean cost on the full 89-task pool, with success-rate standard deviation across three repetitions.Cost is the mean replicate cost in US dollars; cost dispersion across repetitions is not reported.Source: Appendix F · pp. 21–22
  • Table 13 reports per-bundle success rate and mean cost on the 82 tasks with a solvability witness under NIST-derived high, excluding seven blocked-by-design tasks.The seven excluded tasks are build-cython-ext, build-pmars, build-pov-ray, count-dataset-tokens, mailman, nginx-request-logging, and kv-store-grpc.Source: Appendix F · pp. 21–22

Pareto frontier membership

  • GPT-5.6 Sol and Grok 4.5 tie at 79.3% success, but Sol costs $51.67 versus Grok’s $75.64, leaving Grok off the frontier.The frontier is computed under NIST-derived high on the 82-task witnessed pool; the success tie is within replicate variation, so cost decides the ordering.Source: Pareto frontier membership · p. 22

Appendix G: Blocked-Action Analysis

Appendix G defines how blocked actions are counted from verified trial evidence and analyzes their distribution across policy axes and tasks. It also reports associations between blocked actions and outcomes, while cautioning that these associations do not establish causes.

Unit and evidence classes

  • A blocked action is one denied operation, identified through enforcement logs or transcript error signatures, with repeated output and duplicate evidence for the same denial collapsed.The counter does not read verifier outcomes, so the outcome associations are not circular.Source: Appendix G · p. 22

Verified-evidence gate

  • Counts use only verified-complete traces: 6,194 of 9,612 trials overall and 2,051 of 3,204 under NIST-derived high, covering 88 of 89 tasks.Truncated traces cannot establish the absence of blocks, so all totals are lower bounds; truncation is flat across policy conditions but uneven across harnesses and tasks.Source: Appendix G · p. 22

Axes and persistence

  • Under NIST-derived high, egress accounts for 5,250 of 5,800 blocked actions (90.5%), averaging 10.5 denials per affected trial versus 1.4 for privilege.Destination-level linking heuristically assigns 62% of egress volume to repeat attempts, but a re-attempt does not establish intent.Source: Appendix G · p. 22

Concentration across tasks

  • The top five tasks account for 52.8% of blocked-action volume: build-pov-ray (693), break-filter-js-from-html (685), mteb-retrieve (596), protein-assembly (555), and count-dataset-tokens (534).The table buckets tasks by total blocked actions summed over bundles and verified trials under NIST-derived high.Source: Appendix G · p. 22, Table 15 · p. 23

Exposure follows solvability

  • Blocked-action incidence rises from 14.4% on reference-compatible tasks to 51.6% on authored-witness tasks and 67.8% on tasks blocked by design.The respective groups comprise 1,107, 764, and 180 trials; the passage notes the gradient also holds when truncated-trace trials are included.Source: Appendix G · pp. 23–24

Association with outcomes

  • On witness tasks, trials without blocked actions pass at 66.2%, versus 46.7% for trials with at least one; failing trials record 2.7 times as many blocks as passing trials.The authors treat these as diagnostic associations, not causes; harder tasks can be both more blocked and more likely to fail.Source: Appendix G · p. 24
  • On reference-compatible tasks, the pass rate is 80.5% without blocked actions versus 59.1% with them, but passing trials average more blocks than failing trials, 1.04 versus 0.67.The passage characterizes blockage there as incidental and says failures are mostly not blockage-shaped.Source: Appendix G · p. 24
Loading 2608.02670v1…