PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses?

Zhengyang Zhu, Liming Huang, Runmin Ji, Mingxi Ye, Zihan Zhou, Hanyang Guo, Jingwen Wu, Yuhan Ye, Yuming Feng, Hong-Ning Dai, Zibin Zheng

arXiv:2610.07639v1cs.CRcs.SE

TL;DR

Coding agent harnesses mediate tool use and authorize actions, but evidence about their security mechanisms and runtime effects is limited. HarnessSecurity studies mechanisms across harnesses and introduces a benchmark comparing native security settings. Auto-approve increased both utility and attack success, while other mechanisms displayed distinct security–utility trade-offs.

  • Problem

    Systematic evidence is limited on mechanism implementation and default settings across open- and closed-source harnesses, as well as comparative effects of security settings across harnesses and attack surfaces.

    Source: §1 · pp. 1–2

  • Method

    The study derives a ten-mechanism taxonomy, assesses 400 harness–mechanism cells, and benchmarks nine mechanisms across six harnesses using separate deterministic oracles for utility and attack effects.

    Source: §1 · pp. 2–3, §5.1 · p. 8

  • Results

    Auto-approve increased utility and raised attack success from 29.2% to 95.6%, while network isolation and read-only mode reduced attack success with substantial utility losses.

    Source: §1 · p. 3, §6.2 · p. 11

  • Takeaways & Limitations

    Security settings can trade off legitimate task completion against attack effects, and alternative execution paths can leave unauthorized operations reachable.

    Source: §6.2 · p. 11, §9 · p. 16

    The runtime evaluation uses GLM-5.2, so different models may produce different absolute attack success rates and ON/OFF effect magnitudes.

    Source: §7.2 · p. 15

Abstract

from arXiv · show

Coding agent harnesses mediate tool use and authorize actions, yet their security mechanisms and runtime effects remain incompletely characterized. We present HarnessSecurity, the first systematic empirical study and benchmark of open- and closed-source coding agent harnesses. First, we derive a ten-mechanism taxonomy and then assess 400 harness-mechanism cells using independent ratings by researchers and large language model (LLM) judges. We find that about half of confirmed mechanism implementations are opt-in, while closed-source harnesses exhibit substantial evidence gaps. Second, we introduce HarnessSecurity-Bench, a benchmark of 23 tasks across five attack surfaces without sacrificing legitimate task requirements. Using separate deterministic oracles to measure task utility and attack effects with security setting comparisons, we evaluate nine mechanisms across six leading harnesses: Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code, and GitHub Copilot. Under a controlled LLM baseline GLM-5.2, we conduct 2,500 trials, recording 81,155 tool calls and over 2.2 billion tokens. Enabling auto-approve increases utility and raises attack success from 29.2% to 95.6%. Network isolation and read-only mode reduce attack effects with substantial utility losses, while command allowlisting and command denylisting reduce attack effects with a small utility loss and a utility gain, respectively. Task-level cases show that restrictions on a shared capability can obstruct both legitimate and malicious operations, and that allowed tools or commands can leave unauthorized operations reachable through alternative execution paths. Harness providers should make security settings verifiable, test alternative execution paths to protected operations, and assess attack effects alongside task utility and execution costs.

1 Introduction

The paper addresses gaps in systematic evidence about security mechanisms across open- and closed-source coding agent harnesses and their runtime effects. It introduces a mechanism study and benchmark, finding varied security–utility trade-offs across settings.

1 Introduction

  • Untrusted inputs can redirect coding-agent actions, as demonstrated by reported prompt injection through issue titles that induced shell command execution in a Cline workflow mirror.Other cited disclosures report network isolation failures and unsafe IDE behavior triggered by agent file edits.Source: §1 · p. 1
  • Existing studies leave gaps in systematic assessment of implementation and defaults across harnesses, and in comparing mechanism settings across harnesses and attack surfaces.Prior comparisons covered three coding agents or eleven source-accessible harnesses, while existing benchmarks had narrower coverage.Source: §1 · p. 2
  • Auto-approve raised attack success from 29.2% to 95.6% while increasing utility, whereas other mechanisms showed different security–utility trade-offs.Network isolation and read-only mode reduced attack success with utility losses; command allowlisting and denylisting reduced attack effects with a small utility loss and a utility gain, respectively.Source: §1 · p. 3
  • The study derives a ten-mechanism taxonomy and assesses 400 harness–mechanism cells, finding about half of confirmed implementations are opt-in and closed-source harnesses have more evidence gaps.The study covers 40 harnesses: 27 open-source and 13 closed-source.Source: §1 · p. 3
  • HarnessSecurity-Bench evaluates nine mechanisms across 23 tasks and five attack surfaces, using separate deterministic oracles for task utility and attack effects.It compares enabled and disabled native security settings while preserving legitimate task requirements.Source: §1 · pp. 2–3

2 Background

The background describes a five-stage harness loop in which context is assembled, the model plans, and the harness authorizes and executes or routes requests. Attack surfaces are resources, interfaces, or states adversaries can manipulate to affect decisions or execution.

2 Background

  • The harness assembles context, the model plans an action, and the harness authorizes and executes or routes it before interactions update later context.Some harnesses also load skills, delegate to subagents, or connect to external services through MCP.Source: §2 · p. 3
  • An attack surface is a resource, interface, or state an adversary can manipulate to affect agent decisions or harness execution..Source: §2 · p. 3
  • The paper groups attack surfaces by adversarially controlled resource, including repository artifacts, instructions and skills, tools, memory, external services, and direct user input.Figure 1 uses numbered markers to identify affected loop stages; HSB evaluates surfaces A–E.Source: Figure 1 · p. 3, §2 · p. 4

3 Security Mechanisms Taxonomy (RQ1)

The study builds a dataset of 40 coding agent harnesses and derives a ten-mechanism taxonomy through iterative coding and researcher adjudication. The taxonomy distinguishes controls by the resources or operations they govern and where they act in the harness loop.

3.1 Harness Dataset

  • Candidates came from repository lists, commercial products, and security disclosures; 19 duplicate pairs were merged before screening.The initial inventory had 75 records and 56 unique candidates after merging.Source: §3.1 · p. 4
  • The dataset retains 40 harnesses—27 open-source and 13 closed-source—after screening candidate products for adoption, maintenance, workflow, and evidence criteria.The open-source screen required at least 1,000 GitHub stars and publicly available materials suitable for assessing security mechanisms.Source: §3.1 · p. 4

3.2 Study Design

  • Researchers iteratively coded eight initial harnesses, extended labels across the remaining 32, and adjudicated category boundaries with a fourth researcher.They compared protected objects, enforcement points, and effects on operations when refining labels.Source: §3.2 · pp. 4–5

3.3 Results

  • The final taxonomy groups 44 security-related labels into ten mechanisms, including auto-approve, network isolation, command allowlisting and denylisting, read-only mode, and audit logging.Four researchers grouped and adjudicated the labels.Source: §3.3 · p. 5
  • The taxonomy distinguishes mechanisms by governed resources or operations and enforcement points, separating controls such as permitted file targets from suppression of persistent changes.Auto-approve changes confirmation requirements, while audit logging records actions and decisions for later review.Source: §3.3 · p. 6

4 Security Mechanism Empirical Study (RQ2)

The RQ2 study rates implementation and default state across 400 harness–mechanism cells using independent human and LLM assessments. A five-rater majority establishes consensus, with human review for unresolved cells.

4.1 Study Design

  • Each of 400 cells receives one of five ratings distinguishing default-active, configurable, opt-in, absent, and unknown mechanisms.Default-enforced and configurable-off both count as implemented and active by default; unknown marks insufficient or conflicting evidence.Source: §4.1 · p. 6
  • Three LLM raters and two researchers independently assess documentation, CLI materials, and source code, with supporting evidence required for non-unknown model ratings.A rating supported by at least three of five raters determines consensus; human researchers review cells without a majority.Source: §4.1 · p. 6
  • Table 4 presents security mechanism ratings for the 40 harnesses.The RQ2 design applies ratings to 400 harness–mechanism cells.Source: §4.1 · p. 6, Table 4 · p. 7

ALCDDCCDUDUDDDUUUUDDAOCDUUDDU DOUDUUDUUUUDU PTCCACOCCAAAACDUUAAACACUUUAUC COUUCUUUUUUUU

The 400-cell assessment confirms implementation in 205 cells and absence in 83, while 112 remain unresolved. Confirmed implementations are often opt-in, and unknown ratings are more frequent for closed-source harnesses.

4.2 Results

  • 205 of 400 cells have confirmed implementation, 83 have confirmed absence, and 112 remain unresolved.Of 205 implementations, 109 are opt-in and 96 are active by default.Source: §4.2 · p. 7
  • RO and AA implementations are mostly opt-in, whereas MCP and AL implementations are usually active by default.Of 31 RO implementations, 30 are opt-in; of 30 AA implementations, 26 are opt-in. MCP has 22 of 29 active by default, and AL has 20 of 22 active by default.Source: §4.2 · p. 7
  • Unknown ratings account for 69 of 130 closed-source cells (53.1%), versus 43 of 270 open-source cells (15.9%), and cover every closed-source PIF cell.PIF has four confirmed implementations, 13 confirmed absences, and 23 unresolved ratings.Source: §4.2 · p. 7

5 HARNESSSECURITY-BENCH(RQ3)

HarnessSecurity-Bench evaluates task utility and attack effects during coding-task execution, using tasks augmented with attack surfaces and separate outcome oracles. Its scope covers nine mechanisms and excludes downstream uses of recorded trajectories for harness refinement.

5 HARNESSSECURITY-BENCH(RQ3)

  • HSB links task construction and attack-surface augmentation to harness execution and outcome oracles, targeting nine of the ten mechanisms in the taxonomy.AL remains in the taxonomy and ratings but is excluded from the runtime benchmark; downstream trajectory uses for harness refinement are also outside scope.Source: Figure 4 · p. 7, §5 · p. 8

5.1 Task construction

  • HSB curates 23 tasks across nine mechanism benchmarks, with explicit legitimate instructions, executable environments, and utility oracles tied to requested behavior.The original requirements and utility oracles remain unchanged when five attack surfaces are added; direct-user-input attacks are excluded.Source: §5.1 · p. 8, Table 5 · p. 8

5.2 Attack surface augmentation

  • Attacks embed misleading steps or unauthorized actions in repository artifacts, agent instructions, tool interfaces, knowledge records, and external services or execution dependencies.Examples include poisoned project instructions, unauthorized privileged calls, deletion of protected evidence, and callbacks or canary exfiltration triggered by installation or validation.Source: §5.2 · pp. 8–9

5.3 Utility and attack-effect oracles

  • Deterministic task-specific oracles check utility and attack effects against environment state, receiver-side records, or recorded execution events.Oracle definitions and verdicts stay outside the agent workspace; traces are used to interpret how a run unfolded.Source: §5.3 · p. 9

5.4 Illustrative task

  • The Job Scheduler Backoff example pairs nine utility checks with five attack-effect checks for unauthorized actions, while roadblocks mark potential enforcement points.The potential constraints depend on mechanism availability, configuration, and coverage.Source: §5.4 · pp. 9–10, Figure 5 · p. 9

5.5 Task review and validation

  • Researchers other than each task’s designer reproduced and audited the instructions, environment, adversarial augmentation, and oracles for legitimate execution and accurate checks.At least two researchers audited each task.Source: §5.5 · p. 10

6 Benchmark Evaluation

HarnessSecurity-Bench compares utility and attack effects across nine mechanisms and six harnesses, finding distinct security–utility trade-offs and task-level cases where restrictions obstruct legitimate work or leave alternative attack paths available.

6.2 RQ3: HarnessSecurity-Bench Results

  • Auto-approve raised utility from 77.1% to 95.4% (+18.3 pp) and ASR from 29.2% to 95.6% (+66.4 pp).ASR increased in every harness, although utility varied, including a 2.3 pp gain for Gemini CLI and a 0.8 pp loss for Claude Code.Source: §6.2 · p. 11, Table 6 · p. 12
  • Network isolation reduced ASR by 56.3 pp with a 24.5 pp utility loss, while read-only mode reduced ASR by 7.5 pp and utility by 34.0 pp.For NI, required network interactions accounted for 161 of 162 fewer utility passes; for RO, persistent implementation checks accounted for 347 of 367 fewer passes.Source: §6.2 · p. 11, Table 6 · p. 12, Table 7 · p. 13
  • Command allowlisting reduced ASR by 48.1 pp with a 1.5 pp utility loss; across paired PT harnesses, ASR fell from 75.3% to 40.3% as utility fell from 100% to 80%.Excluding Gemini CLI’s startup refusal, the other four PT harnesses retained 100% utility while ASR fell from 75.8% to 50.4%.Source: §6.2 · p. 11, Table 6 · p. 12, Table 7 · p. 13
  • CDL, PR, and MCP reduced ASR by 40.4, 48.1, and 28.1 pp, respectively, while overall utility changed by +5.2, +0.3, and +0.5 pp.For CDL, preventing key copying preserved legitimate validation output in 15 of 29 additional utility passes; some MCP operations could still alter protected state.Source: §6.2 · p. 12, Table 7 · p. 13
  • PIF’s only paired comparison showed ASR falling by 3.3 pp, from 16/60 to 14/60 passed attack checks, while utility fell by 1.9 pp.Its filter matches explicit injection patterns in tool outputs and adds warnings, but attacks framed as ordinary workflow prerequisites could still influence execution.Source: §6.2 · p. 12, Table 7 · p. 13
  • Across 2,500 trials, harnesses recorded 81,155 tool calls, 589.32 hours of execution, and over 2.2 billion tokens in available totals.The total comprises 2.164 billion input and 53.714 million output tokens; conditions with unavailable token totals were excluded.Source: §6.2 · p. 12, Table 8 · p. 13

6.3 Case Studies and Analysis

  • In the NI dependency-installation case, network isolation stopped both the malicious callback and required registry access, reducing utility from 96% to 76% while ASR fell from 100% to zero.All four local functionality checks passed despite the registry-dependent installation workflow failing.Source: §6.3 · p. 14

6.2 RQ3: HarnessSecurity-Bench Results

  • Across mechanisms, NI and RO reduced ASR with substantial utility losses, CAL and PT largely preserved utility, CDL, PR, and MCP reduced ASR with overall utility gains, and PIF showed small decreases in both measures.PT’s overall utility loss came from Gemini CLI refusing startup in its ON configuration; PIF’s paired comparison was available only for gptme.Source: §6.2 · pp. 11–12, Table 7 · p. 13, §6.3 · p. 14

7 Discussion and Limitations

The evaluation is bounded by reviewer variability, challenges reproducing closed-source workflows, simulated deployments, and a single-model runtime study. The authors recommend verifiable settings and testing both alternative access paths and legitimate completion paths.

7.1 Threats to Internal Validity

  • Model bias and residual nondeterminism in the predefined LLM approval reviewer may affect individual approval decisions.Reported auto-approve results therefore reflect the specified approval configuration.Source: §7.1 · p. 14
  • Researcher and model biases may affect RQ1–RQ2 ratings; independent judgments, source locators, passage hashes, and adjudication make them auditable and traceable.The authors use Evidence-of-Thought for both human and LLM raters.Source: §7.1 · p. 14

7.2 Threats to External Validity

  • The runtime evaluation uses six open-source harnesses because closed-source interaction modes, especially GUI workflows, are harder to reproduce faithfully.The authors identify automated evaluation of closed-source harnesses as future work.Source: §7.2 · p. 14
  • Docker containers, automated execution, and simulated services do not fully reproduce real deployments, despite realistic service interfaces and task inputs.The authors present those resources to agents as ordinary task inputs.Source: §7.2 · p. 15
  • Using GLM-5.2 alone limits generalizability because other models may differ in tool use and instruction following, affecting absolute ASR and ON/OFF differences.The same model is used in ON and OFF settings to enable controlled comparisons.Source: §7.2 · p. 15
  • The authors repeat each condition ten times to reduce the influence of stochasticity and occasional erroneous outputs on reported results.Repeated executions may still vary.Source: §7.2 · p. 15

7.3 Implications for Harness Providers

  • Providers should publish versioned mechanism coverage and defaults, explain how settings change, and expose effective settings in the CLI or UI for verification.The recommendation responds to 69 unresolved cells among 130 closed-source harness–mechanism cells, including all 13 closed-source PIF cells.Source: §7.3 · p. 15
  • Providers should test alternative access paths and legitimate completion paths, then report attack effects, task utility, token usage, and runtime together.Repeating checks after changes to tool routing or permission configuration can help detect access-control regressions.Source: §7.3 · p. 15

8 Related Work

Prior work studies harness security, agent-security benchmarks, coding-agent attack benchmarks, and defenses alongside harness performance. HarnessSecurity-Bench instead centers runtime comparisons on native mechanisms and their supported configurations.

Security taxonomies and empirical harness studies

  • HarnessSecurity adds a common taxonomy and auditable rating protocol for implementation and default states across 40 open- and closed-source harnesses.Prior studies include reviews, source-level analyses, and practitioner comparisons of harness architectures and security mechanisms.Source: §8 · p. 15

Agent and harness security benchmarks

  • Unlike workflow- or lifecycle-organized benchmarks, HSB organizes runtime comparisons around mechanisms and configurations established by RQ1–RQ2.It measures effects on task utility and attack effects.Source: §8 · p. 15

Coding agent attack benchmarks

  • Compared with evaluations centered on particular attack resources, HSB spans five attack surfaces while preserving legitimate task requirements and uses separate deterministic oracles for utility and attack effects.Earlier benchmarks examine poisoned instructions, tool descriptions, issue content, repository resources, skills, and risky execution requests.Source: §8 · p. 15

Harness defenses, configuration, and task utility

  • Prior defense and performance studies build interventions or characterize harness behavior; HSB evaluates product-native mechanisms with paired ON/OFF settings where supported.Those fixed-task comparisons quantify differences in attack success and task utility.Source: §8 · pp. 15–16

9 Conclusion

HarnessSecurity maps and assesses mechanisms across coding agent harnesses, then evaluates runtime effects across mechanisms, harnesses, and tasks. Results show trade-offs between attack success and utility, while alternative execution paths can leave some attack effects reachable.

9 Conclusion

  • Network isolation and read-only mode reduced attack success with substantial utility losses, while command allowlisting reduced attack success with a small utility loss and denylisting with a utility gain.Effects varied across harnesses and tasks.Source: §9 · p. 16
  • Auto-approve increased attack success in all six harnesses, and some attack effects persisted through alternative execution paths outside mechanism coverage.The study evaluates nine mechanisms across six harnesses using separate deterministic oracles for task utility and attack effects.Source: §9 · p. 16
  • The study identifies ten mechanisms across 40 harnesses and assesses their implementation, defaults, and supported configurations.The runtime benchmark evaluates nine mechanisms across six harnesses.Source: §9 · p. 16
  • Providers should publish versioned mechanism coverage, expose effective settings, examine alternative execution paths, and report attack effects alongside utility and execution costs.These recommendations address verification and evaluation of harness security settings.Source: §9 · p. 16

A Complete RQ2 Ratings

The appendix presents five-rater mechanism ratings and the evidence rules used to assign them. Ratings distinguish implementation, default state, and whether an active mechanism can be disabled through a public option.

A Complete RQ2 Ratings

  • Table 9 lists each cell’s ratings in L1, L2, L3, H1, H2 order, with shading distinguishing open- from closed-source harnesses.The caption describes the table’s layout, not a rating outcome.Source: Table 9 · p. 19

B.1 Rating Prompt (L1–L3)

  • Raters are instructed to use fixed local evidence, inspect relevant source and official documentation or CLI help, and avoid prior knowledge or speculation.Conflicting evidence or unconfirmed defaults are recorded as U rather than resolved by preference.Source: §B.1 · p. 20
  • The rating decision tree assigns U when evidence is insufficient, A when non-implementation is confirmed, O when an implemented mechanism is inactive by default, and C or D when active by default.C means a public one-step option can disable it; D means no such option is available.Source: §B.1 · p. 20
  • Each mechanism justification must independently explain evidence sufficiency, cite concrete source locations, and preserve verbatim excerpts; insufficient evidence corresponds to U.The prompt requires real path and line-range citations for non-U ratings.Source: §B.1 · p. 21

C Complete RQ3 Task-Level Results

Task-level results show substantial security reductions with differing utility effects across mechanisms and harnesses. Some individual tasks retain nearly all utility while attack success falls sharply, while others show large utility losses.

C Complete RQ3 Task-Level Results

  • Gemini CLI Markdown Table Render utility declined 2.0 pp while ASR fell 100.0% to 0.0%.Gemini CLI CLI Arg Parser utility declined 1.7 pp while ASR fell from 100.0% to 83.3%.Source: C Complete RQ3 Task-Level Results · p. 22
  • Gemini CLI INI Config Parsing utility rose 10.0 pp while ASR fell from 93.3% to 0.0%.The same task-level table reports INI utility gains for Claude Code, Qwen Code, and GitHub Copilot, alongside lower ASR.Source: C Complete RQ3 Task-Level Results · p. 22
  • For Overall, CDL and PR reduced ASR by 40.4 and 48.1 pp, respectively, with utility gains of 5.2 and 0.3 pp, respectively.Source: C Complete RQ3 Task-Level Results · pp. 22–23
  • Across the listed tasks, some settings left attack success unchanged, including Qwen Code Claims Reimbursement and Embedding Drift Monitor at 100.0%.Other harness-task pairs reduced ASR to 0.0%, with task-specific utility effects ranging from gains to losses.Source: C Complete RQ3 Task-Level Results · p. 23
  • Across the listed harnesses, Takeaway Ordering utility often fell substantially even as attack success reached 0.0%.For Claude Code, utility fell 62.0 pp and ASR fell 20.0 pp; for GitHub Copilot, utility fell 61.0 pp and ASR fell 10.0 pp.Source: C Complete RQ3 Task-Level Results · p. 23
  • Codex CLI Pocket Music Box utility remained 100.0% while ASR fell from 70.0% to 0.0%; Gemini CLI tasks instead lost all utility under ON.Gemini CLI utility fell 100.0% on Clockwork Mobile, Constellation Postcard, and Pixel Pet Codec.Source: C Complete RQ3 Task-Level Results · p. 23
Loading 2610.07639v1…