PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent READMEs: An Empirical Study of Context Files for Agentic Coding

Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa, Brittany Reid, Kundjanasith Thonglek, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, Bram Adams, Ahmed E. Hassan, Hajimu Iida

arXiv:2511.12884v2cs.SE

TL;DR

Agent context files are persistent project-level instructions for coding agents, but their structure, maintenance, and content have not been systematically characterized. This study analyzes 2,303 files from 1,925 repositories and finds difficult-to-read, incrementally evolving artifacts dominated by functional guidance, with sparse security and performance instructions.

  • Problem

    Agent context files steer autonomous coding agents, yet the concrete artifacts teams use in repositories remain underexplored.

  • Method

    The study empirically analyzes 2,303 context files from 1,925 open-source repositories across Claude Code, OpenAI Codex, and GitHub Copilot.

  • Results

    Testing appears in 75.9% of files, Implementation Details in 70.8%, and Architecture in 68.1%, compared with Security at 14.8% and Performance at 14.5%.

  • Takeaways & Limitations

    Developers extensively specify how agents should build and operate software, while context files provide comparatively few explicit non-functional guardrails.

  • Takeaways & Limitations

    The snapshot does not track how the semantic content of individual commits evolves over time, including when non-functional instructions are added.

Abstract

from arXiv · show

Agentic coding tools receive goals written in natural language, break them down into specific tasks, and write or execute code with minimal human intervention. Central to this process are agent context files (e.g., AGENTS.md and CLAUDE.md) that provide persistent, project-level instructions. In this paper, we conduct the first large-scale empirical study of 2,303 agent context files from 1,925 repositories to characterize their structure, maintenance, and content. We find that these files are not static documentation but complex, difficultto-read artifacts that evolve like configuration code through frequent, small additions. Our content analysis of 16 instruction types shows that developers prioritize functional context, such as test procedures (75.9%), implementation details (70.8%), and architecture (68.1%). We also identify a significant gap: non-functional requirements such as security (14.8%) and performance (14.5%) are rarely specified. These findings indicate that while developers use context files to make agents functional, they provide few guardrails to ensure that agent-written code is secure or performant, highlighting the need for improved tools and practices.

1 INTRODUCTION

Agent context files provide persistent project-specific guidance for autonomous coding agents, but their real-world structure, maintenance, and instructional priorities had remained underexplored. This study examines these artifacts at scale and finds that they are difficult-to-read, incrementally maintained configurations dominated by functional guidance.

  • The study analyzes 2,303 agent context files from 1,925 repositories across Claude Code, OpenAI Codex, and GitHub Copilot.
  • Agent context files are generally long and difficult to read, while following a consistent shallow hierarchy centered on one top-level heading and H2/H3 sections.
  • Developers maintain these files through short, incremental additions, making them living configuration artifacts rather than static documentation.
  • Instructions emphasize functional operations, whereas Security and Performance requirements are rarely included.
  • Automatic classification reaches 0.79 F1 for concrete topics such as Testing and Architecture but struggles with abstract topics such as Maintenance.

2 MOTIVATING EXAMPLES

Agent context files act as persistent project memory, but their size and organization vary substantially across projects. Existing documentation is limited, leaving developers with fragmented guidance for configuring agent behavior.

  • Agent context files preserve architectural patterns, testing commands, coding conventions, and operational rules across agent sessions.
  • A minimal bioinformatics file contains 21 lines and 3 sections, whereas an agent-tool project example contains 329 lines and 74 sections.
  • Official documentation provides limited guidance beyond sharing project architecture, coding standards, and common workflows.

3 METHODOLOGY

The study systematically collects agent context files from open-source repositories using a repository dataset, star threshold, and filename-based GitHub scanning. The resulting corpus covers the official context-file conventions of three agentic coding tools.

  • The collection pipeline starts from AIDev repositories and retains projects with at least 5 GitHub stars to exclude toy projects.
  • The researchers scan repository roots case-insensitively for CLAUDE.md, AGENTS.md, and copilot-instructions.md using official naming conventions.
  • The corpus contains 922 Claude Code files, 694 OpenAI Codex files, and 687 GitHub Copilot files across 1,925 repositories.

4 RESULTS

The supplied passage introduces the paper’s research-question findings section but does not provide empirical results for Section 4.

  • Section 4 presents the motivation, approach, and findings for each research question.

4.1 𝑅𝑄1: What are the characteristics of agent context files?

Agent context files are generally long and difficult to read, yet they use a consistent shallow Markdown hierarchy; their length differences reflect platform conventions more than task difficulty.

  • Claude Code and GitHub Copilot files are significantly longer than OpenAI Codex files, with median lengths of 485.0, 535.0, and 335.5 words, respectively.
  • Agent context files are generally difficult to read, with Claude Code hardest and OpenAI Codex easiest.The study measures readability using Flesch Reading Ease, where higher scores indicate easier text.
  • Context-file length differences cannot be explained by task difficulty: OpenAI Codex tasks have the highest median cyclomatic complexity but fewer net lines of code than Claude Code and GitHub Copilot tasks.
  • High Flesch Reading Ease scores do not necessarily indicate useful instructions because list-heavy writing can inflate readability scores.Easy files are mainly short, single-clause instructions in flat bullet lists, whereas harder files contain dense architectural prose and technical terminology.
  • Files follow a shallow hierarchy centered on one H1 heading, with typically 6–7 H2 headings and very rare H5 or deeper nesting.The median H1 count is 1.0 across tools, while H5 headers appear in only 9 files and H6 headers are nonexistent.

4.2 𝑅𝑄2: How often do developers maintain agent context files?

Agent context files are actively maintained as evolving configuration artifacts rather than static documentation, with frequent substantive edits made in short bursts and driven mainly by additions.

  • A majority of context files are modified in multiple commits, ranging from 59.4% for OpenAI Codex to 67.0% for Claude Code.Claude Code files receive significantly more commits than GitHub Copilot files, with a small effect size.
  • Substantive instruction edits appear in 92% of 100 sampled post-adoption commits, while purely non-functional maintenance is the only label in just 8%.
  • Updates occur in short bursts, with median intervals of 23.8 hours for Claude Code, 22.3 hours for OpenAI Codex, and 68.0 hours for GitHub Copilot.The day-scale intervals and clustered deletion categories are interpreted as ongoing instruction correction and extension rather than immediate prompt debugging.
  • Deletions remain negligible across file types, with median values below 15.0 words, while Claude Code additions reach a median of 57.0 words per commit.
  • Maintenance is driven mainly by incremental additions: additions appear in 66 of 100 sampled commits, whereas deletion labels appear in only 19.

4.3 𝑅𝑄3: What instructions are included in agent context files?

Agent context files emphasize functional guidance for building, testing, implementing, and maintaining software, while non-functional requirements are substantially less common. They also establish project context, agent responsibilities, and development workflows.

  • 4.3.3 Findings.: Testing appears in 75.9% of files, followed by Implementation Details at 70.8% and Architecture at 68.1%.These categories cover automated-test procedures, coding guidance, and high-level system design.
  • 4.3.3 Findings.: Performance, Security, and UI/UX occur in only 14.5%, 14.8%, and 8.7% of files, respectively.Functional categories therefore substantially outnumber broader quality and user-facing guidance.
  • 4.3.3 Findings.: System Overview instructions occur in 59.0% of files, providing project identity, mechanisms, and key functionality.These instructions supply contextual explanations alongside operational guidance.
  • 4.3.3 Findings.: AI Integration instructions occur in 24.1% of files and define an agent’s role, expertise, and project responsibilities.Examples include guidance for development, content management, SEO, and performance tuning.
  • 4.3.3 Findings.: Build and Run instructions occur in 63.0% of files, while Development Process instructions occur in 65.1%.These categories document commands, workflows, version-control practices, and quality checks.
  • 4.3.3 Findings.: Configuration and Environments appear in 38.9% of files, DevOps in 18.4%, and Project Management in 5.4%.The examples cover environment setup, CI checks, deployment operations, and backlog organization.
  • 4.3.3 Findings.: Maintenance guidance appears in 44.6% of files and addresses readability, bug resolution, and preservation of stable interfaces.The example emphasizes preserving public function signatures, argument positions, and names.
  • 4.3.3 Findings.: Performance guidance appears in 14.5% of files and distinguishes automatic React 19 optimizations from cases requiring manual optimization.Examples include avoiding unstable references and unnecessary object or array creation during rendering.

Instructions directly govern the structural and stylistic properties of agent-generated

Agent context instructions can directly shape generated code, and the study tests this relationship through formatting-rule case studies. It also evaluates automated taxonomy classification, finding strong overall performance but weaker recognition of abstract categories.

  • Instructions directly govern the structural and stylistic properties of agent-generated: The case studies measure machine-checkable indentation compliance before and after adding an explicit context-file rule.Four-space indentation was selected because its intent is unambiguous and every added line can be checked automatically.
  • Instructions directly govern the structural and stylistic properties of agent-generated: The taxonomy contains 16 instruction categories, with functional guidance more prevalent than Performance, Security, and UI/UX instructions.The broader content analysis identifies a concentration on Build and Run, Implementation Details, Architecture, and Testing.
  • 4.4.1 Motivation.: The classification goal is to enable ecosystem-wide monitoring of agent configuration rather than to feed predictions back into coding agents operationally.The motivation is to detect emerging gaps, anti-patterns, and best practices as context files proliferate.
  • 4.4.1 Motivation.: Automatic classification is evaluated as a multi-label task using GPT-5 on 332 manually labeled files and 2,069 label assignments.The prompt includes full file content, category definitions, representative examples, and JSON-structured outputs.
  • 4.4.2 Approach.: The classifier achieves a 0.79 micro-average F1-score and performs well on concrete categories such as Architecture, Testing, Build and Run, and Implementation Details.The evaluation compares predictions with human-established ground-truth labels using precision, recall, and F1-score.
  • 4.4.3 Findings.: Maintenance, Project Management, AI Integration, and Documentation receive comparatively weaker classification performance than concrete technical categories.The study attributes this pattern to abstraction, semantic overlap, or sparse examples.
  • 4.4.3 Findings.: The paper concludes that classification is feasible, but performance varies with how concrete, frequent, and consistently expressed an instruction category is.Architecture and Testing are highlighted as strong examples, while Maintenance is a weaker one.

5 IMPLICATIONS

The findings motivate treating agent context files as evolving configuration artifacts and strengthening their structure, governance, retrieval, and coverage of non-functional requirements.

  • Implications for researchers: Security and Performance instructions are rare, with prevalence of 14.8% and 14.5%, respectively.The authors recommend explicitly incorporating non-functional requirements into agent-facing guidance and benchmarks.
  • Implications for researchers: Low prevalence of non-functional requirements does not necessarily mean developers neglect security and performance outside agent context files.They may rely on code review, external tooling, or later additions, but the study captures only a single snapshot.
  • Implications for developers: Agent context files should be treated like living configuration artifacts, with review processes checking whether code changes require corresponding context updates.Frequent incremental additions risk context drift unless these files are maintained alongside the codebase.
  • Implications for developers: Developers should govern frequent small additions through versioning, changelogs, and approval for high-impact sections.The recommendation responds to the risk that files become unstructured append-only logs.
  • Implications for tool builders: Authoring tools can exploit the consistent shallow hierarchy and common instruction categories to provide scaffolds for context files.Suggested scaffolds include categories such as Build and Run and Implementation Details.
  • Implications for tool builders: Semantic categorization could improve retrieval by prioritizing task-relevant sections, such as Debugging and Testing, over unrelated content.The recommendation is based on the distinct information clusters identified in the instruction taxonomy.

6 RELATED WORK

The paper distinguishes its study of persistent, project-level agent context files from prior work on agent architectures, prompts, documentation, and context strategies.

  • AI agents in software engineering: Prior agent research emphasizes planning, tool use, code generation, repair, refactoring, quality assessment, and multi-agent collaboration.The literature spans agents that execute multi-step tasks and interact with development tools.
  • AI agents in software engineering: Greater agent integration introduces reported integration defects and security risks across agent, connector, vector-store, and execution layers.Related work also examines developer intervention and security risks associated with tool-integration protocols.
  • AI agents in software engineering: This study examines persistent configuration artifacts that steer agents in repositories, a layer largely missing from prior agentic software engineering research.The files are automatically loaded across sessions and accumulate roles, constraints, workflows, and conventions alongside code evolution.
  • Software documentation: Traditional documentation research focuses on developer-facing artifacts such as READMEs, API references, comments, issue discussions, and review discussions.That work studies documentation quality, evolution, inconsistency, and automation.
  • Software documentation: Agent context files differ by targeting AI agents, using imperative constraints, coupling to workflows, and including categories such as AI integration guidance and role definitions.The paper positions these files as distinct from traditional documentation artifacts studied at scale.
  • Context engineering: Context engineering extends prompt engineering toward persistent artifacts, while this paper studies the long-lived files practitioners create rather than experimentally varying context inputs.Prior studies commonly evaluate how prompt or retrieval strategies affect downstream task quality.

7 THREATS TO VALIDITY

The study’s validity is constrained by subjective manual labeling, imperfect proxies for readability and structure, and limited coverage of agentic coding systems and repositories.

  • Internal Validity: Manual classification introduces potential human error and subjective bias, despite independent inspection achieving 80.3% agreement and third-inspector adjudication.The agreement process reduces but does not eliminate concerns about labeling reliability.
  • Construct Validity: The 16-category presence measure captures topic prevalence, not the depth, complexity, or qualitative richness of instructions.A category frequency therefore does not indicate how extensively a topic is addressed.
  • Construct Validity: FRE scores may reflect technical vocabulary and document form rather than developers’ actual comprehension or usability.Imperative directives can appear easy, while technically rich architectural descriptions can appear difficult despite being actionable to domain experts.
  • Construct Validity: Header-based structural analysis captures explicit Markdown headers but misses visually structured sections expressed through bold text, rules, or other conventions.Reported H1–H5 distributions therefore reflect explicit Markdown usage rather than all structural organization.
  • External Validity: Generalizability is limited because the dataset covers 2,303 files from 1,925 repositories using only Claude Code, OpenAI Codex, and GitHub Copilot.The authors identify broader sampling as future work.

8 CONCLUSION

The study provides the first empirical analysis of agent context files, showing that they are actively maintained, difficult-to-read artifacts whose instructions emphasize functional guidance while rarely specifying non-functional requirements.

  • 2,303 context files from 1,925 repositories were analyzed to establish an empirical foundation for understanding their structure, maintenance, and content.
  • Agent context files are generally long and difficult to read, while their shallow heading hierarchy organizes major topics for parsing and maintenance.
  • Unlike write-once documentation, these files evolve through active maintenance and small incremental additions, with deletions negligible.
  • Testing (75.9%), Implementation Details (70.8%), and Architecture (68.1%) are the most prevalent instruction categories.
  • Security (14.8%), Performance (14.5%), and UI/UX (8.7%) instructions are notably infrequent compared with functional guidance.The study characterizes this pattern as extensive guidance on functional execution but limited attention to quality attributes.

A.1 Mapping of Manifest Headers to Taxonomy Labels

Table 6 maps original manual labels and representative headers from three coding tools to the final taxonomy categories used in the qualitative analysis.

  • Table 6 maps original manual labels and representative headers from GitHub Copilot, Claude Code, and OpenAI Codex to the final qualitative-analysis categories.
Loading 2511.12884v2…