PaperlayerView exact revision on arXiv

Source-linked AI summary

Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions

Zimo Ji, Zekai Zhang, Congying Xu, Zongjie Li, Yudong Gao, Shuai Wang, Shing-Chi Cheung

arXiv:2607.02294v1cs.SE

TL;DR

Existing agent benchmarks largely emphasize task completion, leaving behavior on benign but underspecified instructions unclear. UnderSpecBench varies intent, target, and blast-radius cues across grounded DevOps tasks and evaluates boundary violations. Across five agent×model configurations, 55.8–67.8% of acted runs violate at least one boundary, with target ambiguity dominating wrong-target behavior and blast-radius cues barely affecting action.

  • Problem

    Existing agent studies emphasize task completion or adversarial robustness, leaving unclear how agents behave under benign but underspecified instructions and whether they stay within target and scope.

    Source: B. Coding Agents in DevOps · p. 3

  • Method

    UnderSpecBench varies intent clarity, target certainty, and blast radius across 69 grounded DevOps task families and uses deterministic task oracles to assess action boundaries.

    Source: §I · p. 2, C. Test Oracles · p. 4

  • Results

    55.8–67.8% of acted runs violate at least one boundary across five agent×model configurations; target underspecification dominates wrong-target behavior and blast-radius cues have a weak effect.

    Source: §I · p. 2

  • Takeaways & Limitations

    Safe completion, raw capability, and restraint under underspecification are distinct; restraint is a property of the deployed system, and current agents do not reliably provide it.

    Source: §IV · p. 5, §VIII · p. 11

    The benchmark uses autonomous, no-confirmation execution on containerized abstractions, so its rates are a lower-bound stress test rather than predictions of incident rates in gated deployments.

    Source: C. Limitations of UNDERSPECBENCH · p. 10

Abstract

from arXiv · show

LLM coding agents are increasingly deployed to act autonomously on real production infrastructure. They execute shell commands, modify repositories, and call operational APIs. However, completing a task is not sufficient for safety. A wrong action can cause severe consequences. Existing agent benchmarks largely emphasize task completion, leaving open how agents behave under benign but underspecified instructions. We present UnderSpecBench, a benchmark for measuring action-boundary violations in coding agents (i.e., Claude Code, Codex, and OpenCode) on DevOps tasks. UnderSpecBench includes 69 task families grounded in documented incidents, CVEs, or tool behavior and organized across four DevOps capability domains and nine operational control surfaces. To isolate underspecification from task difficulty, each task keeps the same environment and ground-truth safe action while varying the instruction along three axes: intent clarity, target certainty, and blast radius. The resulting 2,208 prompt variants are evaluated with deterministic, side-effect-based oracles that separate Safe Success, Wrong Target, and OverScope outcomes; non-action runs are further classified as clarification, refusal, or deferment. Across five agent x model configurations using OpenCode, Claude Code, and Codex, the evaluation results show that underspecification does not mainly make agents fail; it makes them guess. 55.8-67.8% of runs violate at least one boundary. Target underspecification sharply degrades action quality, while blast-radius cues barely reduce action propensity. These findings show that completion-centric evaluation can overstate safe autonomy and motivate mitigations at the model, harness, and system layer.

I. INTRODUCTION

Operational agents can alter shared or production state, so task completion alone does not show whether they respected the intended target and scope. UnderSpecBench evaluates these boundaries under benign underspecification and finds frequent violations, especially when targets are unclear.

Motivation

  • With reduced human review, configurable permissions can let coding agents execute shell commands, modify repositories, and invoke operational APIs.Claude Code users approve 93% of permission prompts, motivating auto mode as a response to approval fatigue.Source: §I · p. 1

Motivation and gap

  • Existing work largely measures task completion or robustness to adversarial inputs, leaving agent behavior on benign but underspecified instructions unclear.In DevOps, a plausible action can affect the wrong object, environment, or scope.Source: §I · pp. 1–2

Findings

  • 55.8–67.8% of acted runs across five agent×model configurations violate at least one boundary.Figure 1 shows agents act and overstep far more often than they achieve Safe Success.Source: Figure 1 · p. 1, §I · p. 2
  • Target underspecification dominates wrong-target behavior, while blast-radius cues have little effect on agents’ decision to act.Higher action rates can raise apparent task success while also increasing wrong-target and over-scoped behavior.Source: §I · p. 2

Contributions

  • UnderSpecBench varies intent clarity, target certainty, and blast radius across 69 incident-grounded DevOps task families while keeping each environment and safe action fixed.Its deterministic oracles distinguish Safe Success, Wrong Target, and OverScope, and classify non-action as clarification, refusal, deferment, or empty output.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

Coding agents use an LLM and tool-execution loop to make changes through commands, edits, and APIs. In DevOps, existing benchmarks measure task capability but not whether agents stay within the intended target and scope.

A. LLM-based Coding Agents

  • A coding agent’s LLM proposes a command, edit, or API call; the harness executes it and returns results in a repeated loop.Approval modes range from per-action confirmation to autonomous execution, optionally sandboxed.Source: A. LLM-based Coding Agents · p. 2

B. Coding Agents in DevOps

  • DevOps actions can change live production state, where a mis-scoped command may have no clean undo.The paper defines an underspecified instruction as one lacking context needed to act confidently, leaving the operation, object, or scope unstated.Source: B. Coding Agents in DevOps · pp. 2–3
  • Existing operational benchmarks measure whether tasks are completed or services restored, not whether agents stay on the intended target and within scope.Reported production incidents across several vendors motivate the benchmark and empirical study,,,,.Source: B. Coding Agents in DevOps · p. 3

III. BENCHMARK DESIGN

UnderSpecBench builds realistic DevOps task families and varies instruction underspecification while holding the environment, tools, and oracle fixed. Its three axes cover intent, target certainty, and blast radius.

III. BENCHMARK DESIGN

  • UnderSpecBench derives realistic DevOps instruction variants and pairs each task family with a deterministic oracle for checking target and scope boundaries.The benchmark contains 69 tasks organized across four parent capability domains and finer-grained operational surfaces.Source: §III · p. 3, A. Task Families · p. 3

A. Task Families

  • The taxonomy groups tasks across change intake and source configuration, build and verification, release and artifact supply chain, and runtime and reliability operations.It uses nine operational control surfaces, organizing tasks by the immediate object intended for modification.Source: A. Task Families · p. 3, Table I · p. 4
  • Task families are grounded in verifiable incidents, CVEs, reports, or tool documentation that establish destructive semantics and the boundary at risk.Examples include the 2017 GitLab production-database deletion and CVE-2019-5736.Source: A. Task Families · p. 3

B. The three underspecification axes

  • Intent clarity (S) varies how explicitly the action and environment are stated; target certainty (B) varies whether the instruction uniquely identifies the object.At higher B levels, under-determined references match several candidates, requiring the agent to confirm or guess.Source: B. The three underspecification axes · pp. 3–4
  • Blast radius (R) varies whether the same operation affects a contained, low-impact surface or shared production state where errors propagate and cannot be undone.The 4 × 4 × 2 matrix yields 32 variants per task and 2,208 prompts; only instruction wording changes.Source: B. The three underspecification axes · p. 4
  • The three axes correspond to request attributes used for authorization decisions in NIST’s ABAC model, while the user-authorized agent is held fixed.The paper treats action, object, and environment as independently underspecifiable attributes.Source: B. The three underspecification axes · p. 4

C. Test Oracles

UnderSpecBench uses a deterministic, task-specific oracle to judge action boundaries. It compares observed state changes and action traces with the task’s intended and protected object sets.

C. Test Oracles

  • Each task oracle specifies the objects the agent should change, with authorized scope and owner, and the objects that must remain untouched.These are the intended set and the over-scope set, respectively.Source: C. Test Oracles · p. 4
  • Scoring compares before-and-after state changes and action traces against those sets to detect intended actions, wrong or protected targets, and scope overruns.Evidence includes file diffs, command logs, and service-side effects.Source: C. Test Oracles · p. 4
  • Because each oracle is hand-written against the known safe outcome, scoring is reproducible and requires no LLM judge.The resulting verdicts support the benchmark’s boundary-centered metrics.Source: C. Test Oracles · p. 5

D. Implementation

The implementation isolates autonomous agent runs in resettable task environments and combines real tools with deterministic emulation. Boundary metrics distinguish safe execution, wrong targets, excessive scope, and different forms of non-action.

D. Implementation

  • Each prompt runs in an isolated, network-restricted Docker container with per-action confirmation disabled.The preloaded environment includes the task’s repository, files, mock services, and seeded state.Source: D. Implementation · p. 5
  • Real binaries handle lightweight tools, while selected state-changing commands and heavier services use deterministic emulation or mock APIs.The mocks reproduce CLI behavior and validation to keep interactions close to real environments.Source: D. Implementation · p. 5
  • Each task separates the agent-visible runtime from harness-held reset scripts, the 32-variant prompt matrix, and oracle ground truth.The task directory contains the seeded project and shadowed CLIs in runtime/.Source: D. Implementation · p. 5, Figure 5 · p. 5

IV. METRICS

  • Deterministic per-run signals record whether the intended safe action occurred, an unintended target was touched, or scope was exceeded.Safe Success requires the right action and nothing more.Source: §IV · pp. 5–6
  • Overstep Rate aggregates Wrong Target and OverScope, while Under Completion Rate counts runs in which none of the three oracle signals fires.Under Completion is context-dependent because declining to guess can be safe under underspecification and high blast radius.Source: §IV · p. 6
  • Non-action is classified as clarification, refusal, or deferment rather than treated as a single undifferentiated outcome.Clarification asks about target, scope, or intent; refusal declines or aborts; deferment stops without an action or clean question.Source: §IV · p. 6

V. EMPIRICAL STUDY

The empirical study asks how underspecification relates to agents’ action propensity, action quality, and non-action, and how OverScope varies across control surfaces and tasks.

V. EMPIRICAL STUDY

  • RQ1 asks how instruction underspecification affects agents’ likelihood of taking action.Source: §V · p. 6
  • RQ2 asks which underspecification axes are most associated with Safe Success, Wrong Target, and OverScope.Source: §V · p. 6
  • RQ3 asks how often agents clarify, refuse, defer, or stop without an informative response.Source: §V · p. 6
  • RQ4 compares OverScope across DevOps control surfaces, especially bounded-object tasks and tasks mediated by shared control planes.Source: §V · p. 6

A. Experimental Setup

The experiment evaluates five agent–model configurations in autonomous mode across the full prompt matrix for all 69 task families.

A. Experimental Setup

  • Three OpenCode configurations vary the model, while Claude Code and Codex are also tested with models used in same-model OpenCode comparisons.The configurations are OpenCode with claude-haiku-4.5, Codex-5.1-mini, and DeepSeek-v4; Claude Code with claude-haiku-4.5; and Codex with Codex-5.1-mini.Source: A. Experimental Setup · p. 6
  • All runs use full autonomous execution mode.Source: A. Experimental Setup · p. 6
  • Every task family is evaluated over all 32 prompt variants.The benchmark contains 69 task families.Source: A. Experimental Setup · p. 6

B. RQ1: Action Propensity

Target underspecification most sharply degrades acted-run quality, while blast-radius cues barely change action propensity. Intent and target underspecification lower action rates, but agents continue to act.

B. RQ1: Action Propensity

  • 65.5% versus 64.0%: pooled Action Rate is nearly unchanged between R0 and R1, and acted-run quality metrics are also nearly flat across blast radius.For R0/R1, acted-run Safe Success Rate is 42.4% versus 43.0%, Wrong Target Rate 37.3% versus 37.6%, and OverScope Rate 54.9% versus 54.3%.Source: B. RQ1: Action Propensity · p. 6, C. RQ2: Action Quality · p. 7, Figure 6 · p. 7
  • 72.3% to 58.5%: pooled Action Rate falls from S0 to S3, and from 70.9% at B0 to 59.5% at B3, but agents still act frequently.Even at B3, the most cautious system acts on 36.4% of runs and OpenCode+DeepSeek-v4 on 81.3%.Source: B. RQ1: Action Propensity · p. 6
  • Across configurations, OpenCode+DeepSeek-v4 acts in 83.0% of runs and has Safe Success Rate 36.8%, Wrong Target Rate 31.8%, and OverScope Rate 44.4%.Haiku-based systems act about half as often, while OpenCode+Codex-5.1-mini acts in 57.5% of runs.Source: B. RQ1: Action Propensity · p. 6

C. RQ2: Action Quality

  • 50.9% to 29.4%: acted-run Safe Success Rate falls from S0 to S3; Wrong Target Rate rises from 31.9% to 50.4%, and OverScope Rate from 47.3% to 66.3%.The effects of intent underspecification are weaker than those of target underspecification; weak intent wording is recoverable when the target is unique.Source: C. RQ2: Action Quality · pp. 6–7, Figure 6 · p. 7
  • 67.9% to 8.6%: acted-run Safe Success Rate falls as target underspecification rises from B0 to B3, while Wrong Target Rate rises from 9.6% to 75.1% and OverScope Rate from 31.4% to 87.0%.Under high target underspecification, agents infer a plausible object from local context and act rather than confirm the intended candidate.Source: C. RQ2: Action Quality · p. 6, Figure 6 · p. 7
  • Target underspecification is the dominant driver of OverScope once agents act; intent underspecification is weaker and blast radius has almost no effect.When the target is missing, agents guess a plausible object from local context; weak intent wording is more recoverable when the target is unique.Source: C. RQ2: Action Quality · p. 7

D. RQ3: The Meaning of Non-action

Non-action is not a single safety outcome: a stop may clarify or silently leave the task undone, and clarification depends on what is underspecified as well as on the model and scaffold.

Reliability of the non-action judge

  • The disposition split uses an LLM judge only for runs with no action; blind human validation agreed on 85 of 100 cases, with κ = 0.860 over 93 after unclear cases were excluded.Table IV lays out human labels by row and LLM-judge labels by column.Source: D. RQ3: The Meaning of Non-action · p. 7, Table IV · p. 8

Action rate by underspecification axis

  • Action rates fall with intent underspecification S and target underspecification B for most systems, but remain nearly unchanged across blast-radius levels R.Table III reports action-rate percentages over scored runs.Source: Table III · p. 8

Composition of non-action

  • Refusal is negligible, at most 2.5% of scored runs for every configuration; the key distinction is clarification versus a silent failure to complete or ask.The paper describes asking as target-seeking: a missing target presents an answerable question, unlike vague intent or a larger blast radius.Source: D. RQ3: The Meaning of Non-action · p. 8

Ask rate across models and scaffolds

  • Ask rates rise monotonically with target underspecification B, but are non-monotone for intent underspecification S and flat across blast-radius levels R.For Claude Code+Haiku, asking peaks at S2 before falling at S3; rates across R are 42.0% and 47.1%.Source: D. RQ3: The Meaning of Non-action · p. 8, Figure 7 · p. 8

Disposition across models and scaffolds

  • Haiku asks more often than DeepSeek-v4, while the same Codex-5.1-mini asks more under the Codex harness than OpenCode, where stops instead become DEFER dry-runs.The respective Ask rates are 38.3% and 44.5% for Haiku across the two scaffolds, 1.7% for DeepSeek-v4, and 31.8% versus 10.5% for Codex-5.1-mini.Source: D. RQ3: The Meaning of Non-action · p. 8

E. RQ4: OverScope

OverScope differs sharply by control surface: it is lower on bounded objects and higher on shared runtime control planes. The discussion recommends human oversight where those shared surfaces make scope harder to contain.

E. RQ4: OverScope

  • Bounded-object tasks change one determinate resource, while runtime actions pass through shared routing, rollout, alerting, quota, or retention mechanisms that can alter global behavior.The paper attributes the runtime contrast to this structural difference, not to instruction difficulty.Source: E. RQ4: OverScope · pp. 8–9
  • 59.8% on deployment and traffic control and 77.2% on infrastructure, capacity, and observability are the OverScope Rates, versus 14.4–37.6% across seven bounded-object surfaces.Safe Success is 16.6% and 12.6% on the two runtime surfaces, compared with 24–35% on the bounded-object surfaces.Source: Figure 8 · p. 9, E. RQ4: OverScope · p. 9

VI. DISCUSSION & LIMITATIONS

  • The paper frames its findings as guidance for practitioners, ecosystem-level mitigations, and the benchmark’s own limits.Source: §VI · p. 9

A. Lessons Learned

  • Keep a human in the loop for shared control planes, where OverScope reaches 60–77%; the paper considers full autonomy reasonable for bounded-object tasks.It also recommends specifying the target and choosing a harness with a first-class Ask-User affordance.Source: A. Lessons Learned · p. 9

B. Possible Mitigations

The paper proposes calibrated restraint, low-friction clarification, and system-enforced boundaries, while noting limits in benchmark abstraction, coverage, and model snapshots. It positions the work against benchmarks of completion, policy adherence, harmful requests, and operational competence.

B. Possible Mitigations

  • The proposed mitigations span model training for calibrated restraint, harness tools and confirmations that make asking easy, and system-level enforcement for high-risk calls.The paper gives eBPF-based syscall tracing as one example of system mediation.Source: B. Possible Mitigations · pp. 9–10

C. Limitations of UNDERSPECBENCH

  • Containerized abstractions omit production human gating, richer state, and organizational context, so measured rates are a lower-bound stress test of autonomous execution, not incident-rate predictions.Each oracle also encodes one ground-truth safe action, so defensible alternatives may not receive credit.Source: C. Limitations of UNDERSPECBENCH · p. 10
  • Infrastructure, capacity, and observability has only three task families; its rates are trends, not precise point estimates, and temporal, environmental, and policy ambiguity remain outside the three tested axes.Source: C. Limitations of UNDERSPECBENCH · p. 10

VII. RELATED WORK

  • Unlike τ-bench and AIOps work, UnderSpecBench holds the operation fixed while varying instruction specificity; unlike harmful-request studies, it tests benign instructions with latent hazards.It also contrasts with question-answering ambiguity because wrong guesses can change world state, which the benchmark scores with a world-state oracle.Source: §VII · p. 10

VIII. CONCLUSION

  • Agents guess when targets are unclear, barely change caution with blast radius, and overstep most often on live operations that are hardest to undo.The conclusion distinguishes safe completion, raw capability, and restraint under underspecification, describing restraint as a deployed-system property.Source: §VIII · p. 11
  • The authors release UnderSpecBench, its oracles, and its harness, and identify clarification credit and harness guardrails keyed to irreversibility and scope as next steps.Source: §VIII · p. 11

Related papers

Loading 2607.02294v1…