PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Engineering for Agentic AI Coding Tools: An Exploratory Study

Matthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, Sebastian Baltes

arXiv:2602.14690v5cs.SE

TL;DR

The paper addresses the lack of systematic evidence on which configuration mechanisms agentic coding tools offer and how repositories adopt them. It maps mechanisms across five tools and analyzes 2,853 GitHub repositories. Context Files dominate, while advanced mechanisms are uncommon and practices differ across tools.

  • Problem

    The paper asks which configuration mechanisms agentic coding tools offer, which are adopted in open-source repositories, and how they are used.

    Source: §1 · p. 2

  • Method

    The study documents configuration mechanisms across five agentic coding tools and analyzes their adoption in 2,853 GitHub repositories.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Context Files dominate and are often the only mechanism; few repositories adopt Skills and Subagents, and Claude Code repositories use the broadest range of mechanisms.

    Source: §8 · pp. 9–10

  • Takeaways & Limitations

    The findings establish a repository-level baseline and motivate longitudinal studies of configuration practices and controlled studies of whether advanced mechanisms provide measurable benefits over Context Files alone.

    Source: §7 · p. 9

    Artifact-detection heuristics identify configuration presence, not whether the corresponding tool is actively used, and a single labeling run excluded 2,204 unsure cases.

    Source: §6.4 · pp. 8–9

Abstract

from arXiv · show

Agentic AI coding tools increasingly automate software development tasks. Developers can configure these tools through versioned repository-level artifacts such as Markdown and JSON files. We present a systematic analysis of configuration mechanisms for agentic AI coding tools, covering Claude Code, GitHub Copilot, Cursor, Gemini, and Codex. We identify eight configuration mechanisms spanning from static context to executable and external integrations and, in an empirical study of 2,853 GitHub repositories, examine whether and how they are adopted, with a detailed analysis of Context Files, Skills, and Subagents. First, Context Files dominate the configuration landscape and are often the sole mechanism in a repository, with AGENTS$.$md emerging as an interoperable standard across tools. Second, few repositories adopt advanced mechanisms such as Skills and Subagents. Skills predominantly rely on static instructions rather than executable scripts. Third, distinct configuration practices are forming around different tools, with Claude Code users employing the broadest range of mechanisms. These findings establish an empirical baseline for understanding how developers configure agentic tools, suggest that AGENTS$.$md serves as a natural starting point, and motivate longitudinal and experimental research on how configuration strategies evolve and affect agent performance.

1 Introduction

Agentic AI coding tools act autonomously on development tasks, while repository-level configuration lets developers tailor tool behavior and the context assembled for model calls. The paper maps these mechanisms across tools and examines their adoption in open-source repositories.

1 Introduction

  • Agentic coding tools proactively pursue defined objectives by interacting with development environments, project artifacts, external data, and command-line tools.An agent is a goal-directed component that decomposes a user goal into substeps, selects and executes tools, and adjusts its plan in an agent loop.Source: §1 · p. 1
  • Harness engineering customizes the software layers around model calls, which assemble context, expose tool schemas, and manage turn-by-turn state.Context engineering designs the context assembled at runtime; harness engineering customizes the harness itself.Source: §1 · p. 1
  • A configuration mechanism tailors tool and agent behavior to a project or workflow, while an artifact is a file or directory implementing one.Context files such as AGENTS.md or CLAUDE.md provide project-specific information and can be version-controlled for collaborative maintenance.Source: §1 · pp. 1–2
  • The study documents eight mechanisms across five tools and analyzes their adoption in 2,853 GitHub repositories.It examines Context Files, Skills, and Subagents in greater detail; Context Files dominate, while Skills primarily use static resources rather than executable scripts.Source: §1 · p. 2

2 Related Work

Prior work examines context engineering and repository-level context artifacts, but the paper identifies a gap in cross-tool mapping and repository-level analysis of configuration adoption.

2 Related Work

  • Prior context-engineering work addressed context retrieval, management, structured schemas, evolving instructions, and programmable context abstractions.Studies also examined AGENTS.md effects on runtime and token consumption at comparable task completion [16], and Cursor rules [14].Source: §2 · p. 2
  • Earlier empirical work studied repository-maintained context files, their content and maintenance, and 328 CLAUDE.md files in Claude Code projects.Those files describe architecture, build commands, and contribution conventions; architectural concerns dominated the CLAUDE.md analysis, followed by development guidelines, project overviews, and testing.Source: §2 · p. 2
  • Robbes et al. [22] measured coding-agent adoption in 128,018 GitHub repositories, using a coarser configuration categorization than this paper’s eight-mechanism taxonomy.Their analysis focused on adoption rates and AI-assisted commit characteristics rather than configuration mechanisms’ internal structure and co-occurrence.Source: §2 · p. 2
  • No prior work maps configuration mechanisms across tools or studies their adoption patterns across repositories.The paper positions its contribution as a cross-tool perspective.Source: §2 · p. 2

3 Data Collection and Analysis

The study sampled established, actively maintained engineered software repositories, then used README-based classification and file-path heuristics to identify repositories using agentic tools.

3 Data Collection and Analysis

  • 37,249 repositories met initial filters for established, collaborative, actively maintained, OSI-licensed software in the ten most common programming languages.Repositories were required to predate broad tool availability, have at least two contributors, and meet commit and watcher thresholds.Source: §3 · p. 3
  • README-based classification labeled 32,564 of 36,184 repositories as engineered; file-name and path heuristics identified 2,853 repositories using agentic tools.The pipeline excluded non-English README files and repositories labeled non-engineered or unsure; three repositories with only empty configuration artifacts were excluded.Source: §3 · p. 3

4 Configuration Mechanisms (RQ1)

The paper identifies eight repository-level configuration mechanisms across five agentic coding tools, with some mechanisms shared across tools and no tool implementing all eight.

4 Configuration Mechanisms (RQ1)

  • The study selected Claude Code, GitHub Copilot, Cursor, Gemini, and Codex, focusing on their CLI-based agentic interfaces that first appeared in 2025.The selection drew on the 2025 Stack Overflow Developer Survey, with Codex substituting for ChatGPT and Cursor included due to prior software engineering research.Source: §4 · p. 3
  • Table 1 maps each mechanism to repository files or directories; a dash indicates a mechanism unavailable when the tools were checked.Detection used repository-level file and directory heuristics, including .claude or CLAUDE.md for Claude Code and .claude/agents/ Markdown files for Subagents.Source: Table 1 · p. 4, §4 · p. 4
  • AGENTS.md is tracked separately when it appears without tool-specific artifacts because multiple tools support it as a tool-agnostic standard.Some detected files for Copilot, Cursor, and Gemini can apply to both conversational and agentic workflows.Source: §4 · p. 4
  • Eight mechanisms span static context through executable and external integrations; Context Files and Skills are supported by all five tools, but no tool implements all eight.The mechanisms are Context Files, Skills, Subagents, Commands, Rules, Settings, Hooks, and MCP servers.Source: §4 · p. 4

5 Adoption of Configuration Mechanisms (RQ2)

Across 2,853 repositories, Context Files were the dominant and often sole configuration mechanism, while adoption patterns differed by tool and mechanisms often co-occurred in recurring combinations. Most repositories configured a single tool, and Context Files grew faster over time than Skills and Subagents.

5.1 Characterization of Repositories in Dataset

  • 70.6% of repositories adopted a single tool, while 17.3% used AGENTS.md without tool-specific configuration; multi-tool overlap was limited.The dominance of single-tool repositories limited the potential for multi-tool confounding in tool-specific analyses.Source: §5.1 · pp. 4–5

5.2 Distribution of Configuration Mechanisms

  • Context Files were the most frequently adopted mechanism across tools, and most repositories relied on them as their baseline configuration.Across tools, adoption ranged from 61.5% to 100%; most repositories included only a single Context File artifact.Source: §5.2 · p. 5
  • Cursor repositories frequently adopted Rules (72.8%), Gemini repositories used Settings (62.3%), and no other mechanism exceeded 20% for Claude, Copilot, Cursor, or Gemini.Codex had only four repositories, too few to characterize reliably.Source: §5.2 · p. 5

5.2.1 Co-occurrence of Configuration Mechanisms.

  • Settings and Hooks co-occurred positively (Cramér’s V=0.36), while Context Files and Rules appeared together less often than expected (122 observed vs. 216 expected).Recurring combinations reflect shared configuration artifacts and which mechanisms each tool supports.Source: §5.2.1 · p. 5

5.2.2 Adoption of Configuration Mechanisms over Time.

  • Context Files increased continuously over time, whereas Skills and Subagents grew comparatively slowly.Copilot instructions began appearing in 2024 and increased after Copilot’s agentic capabilities were introduced in 2025.Source: §5.2.2 · p. 5

6 Details of Configuration Mechanisms (RQ3)

The RQ3 analysis examines Context Files, Skills, and Subagents, finding widespread Context File adoption alongside limited adoption of the more extensible mechanisms.

6.1 Configuration Mechanism: Context Files

  • 2,586 of 2,853 repositories (90.6%) contained Context Files, usually one or two, with CLAUDE.md, AGENTS.md, and copilot-instructions.md most common.Context File adoption was 88.5–96.2% across languages; copilot-instructions.md led in Java, C#, and C++.Source: §6.1 · pp. 5–6, Figure 5 · p. 6
  • CLAUDE.md commonly appeared first, while AGENTS.md was often added later and became a cross-tool standard; CLAUDE.md pointed to AGENTS.md 301 times, and AGENTS.md received 353 incoming references.Repositories beginning with copilot-instructions.md often later added CLAUDE.md or AGENTS.md.Source: §6.1 · pp. 6–7, Figure 6 · p. 7

6.2 Configuration Mechanism: Skills

  • Only 158 repositories defined Skills, and 514 of 601 Skills (85.5%) contained no resource folders, so Skills predominantly used static documentation rather than executable scripts.Only 29 Skills (4.8%) exceeded the 500-line recommendation.Source: §6.2 · p. 8

6.3 Configuration Mechanism: Subagents

  • The sample contained 450 Subagents across 131 repositories, usually one or two per repository; none used Claude Code’s persistent-memory feature.The average was 3.44 Subagents per adopting repository, with a median of two.Source: §6.3 · p. 8

6.4 Threats to Validity

  • Artifact-detection heuristics measure configuration-file presence, not confirmed tool use, and files for Copilot, Cursor, and Gemini cannot distinguish conversational from agentic modes.The study also notes that a generic AGENTS.md could refer to agents other than software-development tools.Source: §6.4 · p. 8
  • The findings cover engineered open-source GitHub repositories and are a February 2026 snapshot; proprietary settings and variation across application domains remain unexamined.Repository classification used README content, a single labeling run, and excluded 2,204 unsure cases; tool conventions may also have changed since collection.Source: §6.4 · pp. 8–9

7 Discussion

Across five tools and 2,853 repositories, configuration is concentrated in static context files, while advanced mechanisms remain uncommon. The discussion highlights AGENTS.md convergence, tool-specific practices, practical configuration choices, and questions for future research.

7 Discussion

  • Static context mechanisms are widely adopted, while executable and external mechanisms such as Skills with scripts, Hooks, and MCP are rare.The authors characterize open-source harness engineering as mostly context engineering.Source: §7 · p. 9
  • AGENTS.md shows tool-agnostic convergence, but layered context files risk redundant or conflicting instructions and Claude Code lacks native support.CLAUDE.md points to AGENTS.md in 301 cases; AGENTS.md receives 353 incoming references, and 493 repositories use it without tool-specific configuration.Source: §7 · p. 9
  • Advanced mechanisms remain limited; 85.5% of Skills lack additional resources, and no repositories use Claude Code Subagents’ persistent memory feature.Among Skills with resources, static references are as common as executable scripts, so Skills function primarily as structured text.Source: §7 · p. 9
  • The authors suggest that advanced mechanisms’ novelty and configuration effort may help explain their limited adoption, while whether deeper configuration improves performance remains to be assessed.Executable Skills with scripts and structured resources require more design and maintenance effort than Markdown files.Source: §7 · p. 9
  • Configuration practices differ across tools: Claude Code repositories use the broadest range, Cursor emphasizes Rules and Commands, and Copilot rarely extends beyond Context Files.Whether these practices converge as tool capabilities overlap remains an open question.Source: §7 · p. 9
  • The authors suggest Context Files, particularly AGENTS.md, as a simple starting point and recommend a shared core with hierarchical tool-specific adapters when files coexist.They also note that tool-specific Rules or Commands tie workflows to that tool.Source: §7 · p. 9
  • For recurring, well-defined workflows, Skills can bundle scripts and structured resources, although most Skills currently lack such resources.The authors propose longitudinal studies and controlled tests of whether Skills or Subagents provide measurable benefits over Context Files alone.Source: §7 · p. 9
  • The findings are a point-in-time snapshot, so identified trends are early empirical signals rather than settled findings.The authors also call for research on detecting and resolving conflicts between overlapping instructions across files.Source: §7 · p. 9

8 Conclusions

The study analyzes repository-level harness configuration across five tools and finds that context files dominate, advanced mechanisms are rarely adopted, and practices differ by tool. It recommends AGENTS.md as a starting point and calls for research on richer configurations and evolving practices.

8 Conclusions

  • Context Files dominate and are often the only mechanism present; AGENTS.md is used across tools even when a developer’s primary tool lacks native support.The study identifies eight mechanisms across five tools and analyzes 2,853 GitHub repositories.Source: §8 · p. 9
  • Few repositories adopt Skills and Subagents; most Skills lack scripts or other executable resources, and no repositories use Claude Code Subagents’ persistent memory feature.Source: §8 · p. 9
  • Practices differ by tool: Claude Code repositories use the broadest range of mechanisms, Cursor emphasizes Rules and Commands, and Copilot rarely extends beyond Context Files.Source: §8 · p. 10
  • The authors identify AGENTS.md as a natural starting point, recommend improved onboarding for Skills and Subagents, and call for controlled and longitudinal research.Research questions include whether richer configuration improves outcomes over Context Files alone and how patterns evolve as tools mature.Source: §8 · p. 10

Related papers

Loading 2602.14690v5…