Scanning the Harness: Configuration Exposures in AI Coding-Agent Supply Chains

1 Introduction

• Harness artifacts distribute instructions, executable dependencies, and permissions through repositories and marketplaces, making configuration an actionable part of the software supply chain.Prior studies examine skill-marketplace vulnerabilities and susceptibility to injected skill instructions [3] [4] [5].Source: §1 · p. 1
• The analysis combines deterministic scanning, mechanical re-derivation, model-assisted adjudication, human review, and documentation checks across 3,171 repositories.The six categories cover security exposures, a required agent field, and skill-specification issues.Source: §1 · p. 1
• 409 of 2,660 setups (15.4%) contain an unpinned MCP package declaration or broad execution-preapproval declaration.Among setups with MCP configuration, the unpinned-declaration rate is 24.5%; required-field and skill-format issues bring the setup rate to 17.9%.Source: §1 · p. 1
• A documented wildcard-find permission exception exposed a shared scanner and audit error, while category selection explains most of the rate reduction from the broader candidate set.Correcting the exception removes 62 findings.Source: §1 · p. 1
• The study contributes reproducible measurement, an accounting of category selection and review, and practical guidance on dependency pinning, permission review, and component conformance.Cross-file and graph alerts complement file-scope detection by indicating where contextual or runtime analysis may be needed.Source: §1 · p. 1

Background

• Harness components include persistent context, task-specific skills, commands, lifecycle hooks, subagent files, and MCP declarations, but loading and permission semantics vary by client.Recognizing a file format alone does not establish its behavior in every client.Source: §2 · pp. 1–2

Skill and agent security

• Prior skill and agent-security work includes marketplace checks and malicious-skill analysis [3] [4], injected-skill benchmarks [5], and runtime-composition or attack evaluations [6] [8] [9] [10].Skill-Inject [5] measures agent responses to injected skills rather than marketplace prevalence.Source: §2 · p. 2

Configuration and harness evaluation

• Harness studies examine context-file contents and effects [14] [15] [16] [17], optimize harnesses from execution traces [18], or evaluate skills through paired execution [2]; this paper measures configuration exposure and conformance across component types.Infrastructure-as-code and configuration-smell studies provide precedent for structural checks [11].Source: §2 · p. 2

Analyzer evaluation

• Analyzer research highlights the difficulty of producing useful developer findings, while mechanical agreement and human–pipeline agreement measure consistency rather than precision against independent ground truth.The paper distinguishes agreement between implementations from agreement between review judgments and pipeline labels.Source: §2 · p. 2

3 Study Design

• The study asks how often selected conditions occur, how rates vary by repository stratum and review policy, and which claims static analysis can support.These questions distinguish measured repository conditions from claims needing runtime or contextual evidence.Source: §3 · p. 2

3.1 Corpus and Units of Analysis

• The discovery funnel leaves 3,171 repositories from 9,295 candidates, classified as 2,660 setups and 511 skill collections; the corpus excludes instruction-only and sparse repositories.Repositories are the unit at pinned commits, counted at most once per rule and union; deduplication does not establish independence among copied configurations.Source: §3.1 · p. 2

3.2 Instrument and Analysis Scope

• The analysis considers 31 candidate rules across security exposure, configuration integrity, specification conformance, and cross-assistant consistency, using file, filesystem-context, and component-pair checks.The taxonomy describes what the implemented checks read, not proof that they completely capture the underlying security property.Source: §3.2 · p. 2

3.3 Exposure Definitions and Evidence

• A security exposure is a declaration leaving a dependency version unspecified or requesting broad execution preapproval; the study does not assume the grant is accidental or exercised.Evidence is separated into declaration existence, documented client meaning, and actual deployment; the study does not measure the last layer.Source: §3.3 · p. 2

3.4 Mechanical, Model-Assisted, and Human Review

• Mechanical re-derivation checks scanner predicates, while contextual adjudication retains disputed repository-rule pairs only when the model labels them defects.Both implementations came from the study team, so agreement can reveal discrepancies but cannot rule out shared errors; model labels are not ground truth.Source: §3.4 · p. 3
• Human review matched 743 historical pairs and agreed with pipeline labels on 93.3% (κ=0.76), versus 70.7% (κ=0.23) for 157 pairs requiring model adjudication.Six rules met the original primary-tier thresholds; these descriptive selection rules are not calibrated deployment guarantees or a preregistered detector test.Source: §3.4 · p. 3

3.5 Semantic Correction and Reanalysis

• Applying Claude Code’s documented wildcard-find exception removes 36 settings findings and 26 skill findings, and the reanalysis records the semantic correction and revised counts.Reported proportions describe the corpus; population confidence intervals are omitted because discovery was not probability sampling and repositories can share components.Source: §3.5 · p. 3

4.1 Corpus Composition and Findings

• 475 setups (17.9%) had one or more of the six reported findings, including 409 setups (15.4%) with a security exposure; categories overlap.Collections had 35 repositories with findings (6.8%). The median setup contained six components and an estimated 7,057 tokens; these are inventory estimates, not measured model context.Source: §4.1 · p. 3, Figure 1 · p. 4

4.2 Dependency and Permission Declarations

• 260 setups (9.8%) had unpinned MCP packages, 67 (2.5%) broad execution grants, and 101 setups (3.8%) broad skill-tool preapproval.The unpinned-package rate was 24.5% among setups with MCP configuration; settings grants and skill preapproval together occurred in 165 setups (6.2%).Source: §4.2 · p. 4

4.3 Configuration and Specification Issues

• 22 setups (0.8%) lacked a required subagent description, while selected skill-format deviations affected 63 setups (2.4%).Skill deviations included missing frontmatter and present frontmatter without a description; the setup union is 63 because one repository appeared in both categories.Source: §4.3 · p. 4

4.4 Separating Rule Selection from Review

• 25.5% of setups were flagged across 31 candidate categories, versus 17.9% after selecting six categories, correcting a wildcard-find interpretation, and applying contextual review.Selection removed 175 repositories from the raw union, compared with 15 for the find correction and 13 for historical contextual decisions; these order-dependent marginal reductions are not independent causal effects. A disputed-pair sensitivity check ranged from 17.8% to 18.3%.Source: §4.4 · p. 4

4.5 Comparing Repository Strata

• Collections structurally lack MCP and hook/settings components, so their zero counts for those declarations do not indicate greater safety.Skill preapproval was 5.1% among setups containing skills, compared with 3.7% of collections; establishing transfer between strata would require matched content or installation histories.Source: §4.5 · p. 5

4.6 Contextual Observations and Graph Alerts

• Contextual and graph alerts exposed interpretation limits: none of six credential-to-network alerts established the claimed exfiltration path, and recall remains unmeasured.Contextual review treated differences in instruction files, missing paths, and apparently incomplete MCP declarations as case-specific. An exploratory reference-resolution rule flagged 33.1% of setups and 64.0% of collections; sampled alerts distinguished dead paths, wrong paths, and planned outputs.Source: §4.6 · p. 5

5 Implications

• The 15.4% result identifies 409 setups with dependency-selection or execution-preapproval decisions to review; 24.5% of MCP-bearing setups have an unpinned declaration.The reported categories can be checked at file scope before running an agent.Source: §5 · p. 5
• Under a threat model involving untrusted instructions or changing dependencies, review execution grants and package resolution before portability warnings.Priority depends on activation, sandboxing, credentials, and overriding policies; the study does not estimate incident likelihood or severity.Source: §5 · p. 5
• Versioned behavioral cases should test broad grants and documented exceptions because scanner and audit implementations can share a client-semantics error.The documented find exception removed 62 findings; project/local bypassPermissions and dontAsk require distinct interpretations.Source: §5 · p. 5
• Cross-file differences and referenced output paths may be intentional, so contextual diagnostics should be separated from automatic fixes.Human assessment of unflagged cases and runtime evaluation are needed to measure recall and consequences.Source: §5 · p. 5

6 Threats to Validity

• The study does not measure whether broad capabilities are unwanted or whether exposures become exploits; deployment policies, trust, activation, and attacker control remain unmeasured.A broad grant need not be unwanted, and a specification deviation need not cause failure.Source: §6 · p. 5
• The checks omit some code-executing tools and Git grants, selected skill requirements, and transitive dependency resolution; remaining permission patterns are not exhaustively tested.Documented semantics can also vary by client version and operating system.Source: §6 · p. 6
• One author performed manual review, and human–pipeline agreement measures neither inter-human reliability nor held-out detector accuracy or deployment consequences.The disagreement sensitivity band covers recorded disputes, not shared errors; unflagged cases must be reviewed to assess recall.Source: §6 · p. 6
• Discovery favors visible, advertised configurations, and corpus proportions are not weighted by users, downloads, skills, or installations.Private enterprise configurations may differ, so the paper makes no ecosystem-wide sampling claims or confidence intervals.Source: §6 · p. 6
• Pinned inputs permit count reproduction but do not establish an identical fresh scan; historical corrections and rule selection used the same corpus.No held-out detector evaluation, runtime exploit study, or installation-provenance analysis was conducted, leaving false negatives and residual false positives unknown.Source: §6 · p. 6

7 Conclusion

• 475 of 2,660 setups (17.9%) and 35 of 511 skill collections (6.8%) contain a reported finding across the measured categories.The 409 setups with unpinned MCP declarations or broad execution preapproval comprise 15.4% of setups.Source: §7 · p. 6
• Correcting a documented permission exception changed counts despite agreement between the scanner and mechanical audit, underscoring the need for explicit semantic validation.The paper identifies fixed-rule accounting, versioned client semantics, and traceable evidence as ways to make measurements reproducible and conclusions assessable.Source: §7 · p. 6

Data Availability

• The scanner and original corpus, audit, adjudications, and reader table are available through the listed public GitHub repositories.The corpus commit is identified as 217b8620e8a7, with the full identifier in the supplement.Source: Data Availability · p. 6
• The revision supplement provides reanalysis and figure scripts, input hashes, revised counts, pair and exclusion tables, reproduction instructions, and a proposed validation protocol.Source: Data Availability · p. 6
• The reader directory preserves human verdicts and agreement statistics, restores original review-column names, and records confirmation of their provenance.One credential-like literal is redacted from scanner evidence; the public commit remains the immutable input baseline.Source: Data Availability · p. 6

• No repository-provided code or exploit was executed for this revision.Source: Ethical Considerations · p. 6
• Repository identities and commit links remain in the artifact, but revised materials reproduce no credential values.Source: Ethical Considerations · p. 6
• Original secret-related alerts require separate handling before redistribution, and no external human reviewers were recruited.Source: Ethical Considerations · p. 6

• Table 5 separates finding-level re-derivation from repository-level inclusion.Source: A Mechanical Agreement for the Six… · p. 7
• The settings and skill categories exclude 62 find findings.Source: A Mechanical Agreement for the Six… · p. 7
• A positive pair can contain several findings, so a high agreement rate does not establish runtime correctness.Source: A Mechanical Agreement for the Six… · p. 7
• Historical broader-review counts refer to the original table, not a new review of the corrected categories.Source: A Mechanical Agreement for the Six… · p. 7