PaperlayerView exact revision on arXiv

Source-linked AI summary

What Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.md

Pawel Zarzycki

arXiv:2610.31715v2cs.SEcs.AI

TL;DR

Projects write agent instructions because coding agents may lack the history through which human contributors learn project norms, yet what popular projects put in those files has not been established. This paper censuses repositories and analyses their instructions at statement level. The results show that instructions combine project knowledge with expectations, while attribution rules differ across collaboration settings and many directives lack explanations.

  • Problem

    Existing studies do not establish what popular projects write in agent instructions statement by statement, where the text comes from, or what it varies with.

    Source: §1 · p. 1

  • Method

    The paper censuses root AGENTS.md and CLAUDE.md files in 22,854 repositories and analyses their instructions using model-assisted annotation and human review.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    86.2% of instruction sets contain project knowledge, and 54.3% contain rules in that area; AI attribution rules differ between commit messages and pull requests.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    Agent instructions combine project context with expectations, but most directives give no reason and projects have no shared norm for AI attribution.

    Source: §6 · p. 10

    Model-derived categories diverge from the human reviewer’s subject assignments, and the review covers only 60 sampled statements from validation files.

    Source: §5 · p. 9

Abstract

from arXiv · show

Coding agents can be guided with instruction files in a repository. We examine this emerging practice through a census of 22,854 popular, active GitHub repositories in thirteen languages, treating each repository's root 'AGENTS$.$md' and 'CLAUDE$.$md' as one instruction set. Agent instructions appear in 26.3% of repositories and 59.6% of the most-starred; where both files exist, 'CLAUDE$.$md' usually points to 'AGENTS$.$md'. We analyse 341,852 statements using model-assisted annotation and human review. The annotations reveal a practical mix of project knowledge and rules about code, building and workflow. Collaboration rules name concrete requirements, from test plans to commit formats, yet 74.0% of directives give no reason. Projects disagree on AI attribution: commit-message rules more often prohibit it than require it, while pull-request rules more often require disclosure. A third of instruction sets have a generator-associated signature, a tool-inserted block, shared text or an AI-marked creation commit; four in five are edited after creation. Prespecified tests associate repository popularity with rules on pull requests and AI contributions. Exploratory analyses link activity to process-oriented content and show that LLM and agent projects are over-represented among adopters. The models agree much more closely with each other than with the human reviewer's topic assignments, a caution for automated annotation. We release the annotations, taxonomies and pointers to the analysed file versions.

1 Introduction

Existing studies do not establish what popular projects write in agent instructions, statement by statement, or where that content comes from and what it varies with. This study addresses those questions through a census and releases annotations, taxonomies, and file-version pointers.

1 Introduction

  • Whether instruction files help agents is contested, and existing studies do not establish what popular projects write, statement by statement, or what the content varies with.The paper frames the gap around files’ content, provenance, and variation.Source: §1 · p. 1
  • The census examines 22,854 public GitHub repositories across thirteen languages, treating each repository’s root AGENTS.md and CLAUDE.md as one instruction set.It asks who has instructions, what they contain, how they are expressed, where content comes from, and what it varies with.Source: §1 · p. 1
  • The authors release annotations, taxonomies, aggregate results, and file-version pointers without redistributing corpus texts.The release supports examination of requirements and traces of generators, tools, and shared text.Source: §1 · pp. 1–2

2 Related work

Prior work studies agent instructions, their effects, rule taxonomies, AI attribution, and repository meta-files. This paper distinguishes itself through a population-wide analysis of popular, active repositories and a codebook that captures how statements instruct.

2 Related work

  • This census differs by studying popular, active repositories not selected for agent use and analysing each repository’s instruction files together at statement level.Earlier work includes studies of agent-selected repositories, individual file types, file histories, and configuration mechanisms.Source: §2 · p. 2
  • Controlled studies of instruction-file effects report mixed findings [Gloaguen et al., 2026, Lulla et al., 2026, Khatri, 2026], while the authors make no effect claims.They measure features named in prior work, including prohibitions, concrete requirements, rationale, and conditions.Source: §2 · p. 2
  • The codebook adds dimensions for how statements instruct, including binding force, conditions, rationale, addressee, actionability, and required collaboration elements.Other taxonomies classify topics, levels, polarity, security rules, and values guidance.Source: §2 · p. 2
  • Mining studies warn that commit trailers and pull requests provide incomplete, easily misread traces of agent activity [Robbes et al., 2026a,b, Khosravani and Mockus, 2026, Yu et al., 2026].This paper examines attribution policies written in instructions for agents.Source: §2 · p. 2
  • The study follows research on repository meta-files and text reuse, while language-model classification can show only fair agreement with human coders.Prior work also studies prompts in code and the spread and divergence of templates [Pister et al., 2024, Hata et al., 2021].Source: §2 · p. 2

3 Data and methods

The study censuses root agent-instruction files in popular, recently active GitHub repositories, combines files into repository-level instruction sets, and analyses statements with model-assisted coding and human review. It also measures document features and applies prespecified tests and exploratory models to relate instruction content to project characteristics.

3.1 Population, census and collection

  • The census covers 22,854 repositories with at least 1,000 stars and recent activity across thirteen languages, checking default-branch root AGENTS.md and CLAUDE.md files.Repositories were enumerated through GitHub’s search API; files, metadata, activity, and histories were collected through GraphQL and REST APIs without cloning.Source: §3.1 · p. 2
  • The unit is a repository’s combined root AGENTS.md and CLAUDE.md text, with symbolic links and thin imports resolved to avoid counting text twice.Structural features use 5,998 joined documents; content is coded per file and pooled per repository.Source: §3.1 · p. 3

3.2 Document structure and an existing taxonomy

  • The study measured document structure and signatures, and applied 16 instruction types from Chatlatanagulchai et al. [2026], calibrated against 324 human-labelled files.The comparison taxonomy was corrected for over-labelling.Source: §3.2 · p. 3

3.3 Statement extraction and annotation

  • A codebook with 10 dimensions and 48 values was developed through model-assisted coding, and Haiku segmented files into 341,852 statements, 99.4% with verbatim spans found.Statements include a restatement, subject, and elements; directives have binding force, while descriptions and plain references are not directives. Human review covered 60 sampled statements and found limited agreement on subject categories, without validating every later category or stance assignment.Source: §3.3 · p. 3
  • Sonnet coding of a stratified sample supplies headline estimates; Sonnet and Opus agree closely, while Haiku agrees less with Sonnet.The sample estimates use 992 repositories with inclusion-probability weights and a repository-cluster bootstrap; element tables and correlations use uncorrected Haiku codes for all files.Source: §3.3 · p. 3

3.4 Prespecified tests and exploratory associations

  • Prespecified hypothesis tests and exploratory associations relate repository-level content to project characteristics using negative-binomial count models and logistic models for indicators.The tests covered 5,587 repositories with at least five coded statements; exploratory analyses covered 5,586 instruction sets and compared adopters with 4,699 matched repositories without agent files.Source: §3.4 · pp. 3–4

3.5 Language models as instruments

  • Language-model coding and classification used Claude Haiku 5.5, Sonnet 5.5, and Opus 5.5 through Amazon Bedrock in October 2026.Structured outputs were constrained to the codes, and prompts instructed models to treat files as data; statement extraction and annotation cost $195.68.Source: §3.5 · p. 4

4 Results

Across the 22,854 repositories, agent instructions are more common among the most-starred projects; their files mostly combine project knowledge with rules about code and workflow. The rules specify concrete collaboration requirements, but many give no reason, and the observed associations vary with repository characteristics.

4.1 RQ1: Who has agent instructions?

  • 26.3% of 22,854 repositories had a root AGENTS.md or CLAUDE.md, rising to 59.6% of the 89 most-starred repositories; where both files exist, CLAUDE.md usually points to AGENTS.md.The adoption share rose from 23.2% in the lowest star tier; 1,540 repositories had CLAUDE.md defer to AGENTS.md, versus 182 with the reverse direction.Source: §4.1 · p. 4, Figure 1 · p. 5

4.2 RQ2: What do agent instructions contain?

  • Code conventions and quality (74.4%), build and delivery (72.6%), and agent workflow (67.3%) were the leading rule subjects; 86.2% of instruction sets contained project knowledge.98.2% contained at least one rule; rules made up an average 52.6% of statements, alongside 29.8% descriptions and 17.6% references.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
  • Collaboration rules named concrete requirements, including test plans for pull requests and Conventional Commits for commit messages, while AI-attribution policies differed by context.Among repositories with commit-message rules, 7.5% prohibited AI attribution and 59 required it; for pull requests, 5.8% required disclosure and 1.9% prohibited it.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3 RQ3: How are agent instructions expressed?

  • 74.0% of directives gave no reason, although 82.9% of instruction sets gave a reason at least once; mandatory rules and prohibitions appeared in most sets.Prohibitions averaged 15.8% of a repository’s directives, while unmarked imperatives were the most common form.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4 RQ4: Where does the content come from?

  • 33.1% of instruction sets with file histories carried a textual trace or were created in an AI-marked commit, and 81.8% were edited after creation.Generator signatures associated with an /init command appeared in 15.8%; these traces indicate reuse or assistance, not whether whole files were human- or machine-written.Source: §4.4 · pp. 6–7

4.5 RQ5: What does content vary with?

  • In prespecified tests, more-starred repositories more often had pull-request or AI-contribution rules (OR 1.36 per standard deviation of log stars), while compiled-language repositories more often had build instructions (OR 1.68).CLAUDE.md was more descriptive than AGENTS.md in the same repository; three of six prespecified hypotheses were supported.Source: §4.5 · pp. 7–8, Table 2 · p. 8
  • Exploratory analyses associated activity with more process-oriented content, while LLM, generative-AI, or agent projects were 27.9% of adopters versus 13.3% of matched controls.The project-type association remained positive after controlling for age and activity and excluding agent-tooling projects, but project-type contrasts were indicative because human-reviewer agreement was limited.Source: §4.5 · p. 8, Figure 4 · p. 9

5 Threats to validity

The study’s validity is bounded by what its data and measures capture, by model-assisted annotation, and by its cross-sectional sample and design. These constraints limit how directly its findings can be generalized or interpreted.

Construct validity

  • The study measures root AGENTS.md and CLAUDE.md files, not agent use or runtime instructions, and excludes nested agent files.AI-marked commits capture only some AI-assisted work, while project type relies on self-written metadata that agrees only moderately with human readings.Source: §5 · p. 9

Measurement

  • Content categories are model-defined, and models agree much more closely with one another than with the human reviewer, including on broad topics.Human review covers a limited sample and excludes unreadable languages; its acceptance rates therefore do not establish corpus-wide accuracy.Source: §5 · p. 9

Internal validity

  • File histories omit files that do not survive to the snapshot and do not follow renames; correlations are cross-sectional, and activity counts may include agent commits.Claude Code began reading AGENTS.md four weeks before the snapshot, which may affect recent shares.Source: §5 · p. 9

External validity and inference

  • The findings concern popular, recently active repositories in thirteen languages on one snapshot date, and exploratory results are leads rather than confirmatory evidence.With almost six thousand repositories, small exploratory effects can become significant; false discoveries are controlled only within each family.Source: §5 · p. 9

6 Discussion

The discussion characterizes agent instructions as evolving onboarding documents: they combine project context with work expectations, while exposing unsettled practices around explanation, reuse, and AI attribution. It also highlights limits in annotation agreement and identifies questions for future study.

Instructions as onboarding with expectations

  • Agent instructions serve as compact onboarding, pairing project context with expectations such as verification, testing, and code-design practices.Project knowledge appears in 86.2% of instruction sets, with rules on that area in 54.3%.Source: §6 · pp. 9–10

Concrete, but mostly unexplained

  • Collaboration rules specify requirements such as test plans, linked issues, and commit formats, but most directives give no reason.Whether reasons or emphasis change what agents do remains open; concrete instructions of this kind are reported to carry most measured gains in agent skills [Wang et al., 2026].Source: §6 · p. 10

Reused, assisted and maintained

  • A third of instruction sets show generator-associated signatures, tool-inserted blocks, shared text, or AI-marked creation commits, while four in five are later edited.A file can combine tool-supplied text, reused guidelines, and project edits, complicating comparisons of developer-written and generated files [Gloaguen et al., 2026].Source: §6 · p. 10

One instruction set

  • Repositories with both file names usually link CLAUDE.md to the substantive AGENTS.md, so counting both independently would duplicate many repositories.Resolving links and imports identifies what the agent is asked to read.Source: §6 · p. 10

AI attribution has no shared norm

  • Commit-message rules more often prohibit AI attribution than require it, whereas pull-request rules more often require disclosure, mostly in other projects.Trailer-based AI-use measures can therefore leave different traces for projects with equal AI use.Source: §6 · p. 10

What content varies with

  • Exploratory models associate activity with more tested content outcomes, while LLM and agent projects remain over-represented among adopters after adjustment for other listed characteristics.The cross-sectional data cannot establish direction, and adoption may shape how projects describe themselves.Source: §6 · p. 10

A method lesson

  • Models agree more closely with one another than with the human reviewer, especially on topics; agreement is higher for distinguishing descriptions from rules.Agreement among models alone offers limited assurance that categories match human interpretation.Source: §6 · p. 10

Future work

  • Future work can test whether repositories follow their own instructions, including AI-attribution rules, and confirm exploratory associations on fresh data.Source: §6 · p. 10

7 Conclusion

The conclusion presents a census of root agent-instruction files as a basis for studying a practice that combines project knowledge with work expectations. It also identifies unsettled norms, reuse and editing, and questions for future research.

Conclusion

  • One in four repositories in the census of popular, active GitHub projects in thirteen languages has root instructions for coding agents.Source: §7 · p. 10
  • The files place project knowledge beside expectations about code, testing, and collaboration, while projects disagree about AI attribution and most directives offer no explanation.Source: §7 · p. 10
  • Exploratory analyses associate activity with process-oriented content and find LLM and agent projects over-represented among adopters.The released annotations, taxonomies, and pointers to analysed file versions provide a basis for studying how the practice develops and whether agents follow written expectations.Source: §7 · p. 10

AI-use statement

The author reports using Claude models for annotation and classification, with coding and drafting assistance from Claude Code, Codex, and GPT models. The author states responsibility for the study’s decisions and content.

AI-use statement

  • Claude models supported segmentation, annotation, project classification, and taxonomy development; Claude Code, Codex, and GPT models assisted with coding, drafting, revision, and review.The author designed the study, made analytic decisions, reviewed and updated code, edited the text, and takes responsibility for its content.Source: AI-use statement · p. 11

A.1 Method details

The method calibrates model-labeled instruction content against reference annotations. It uses different corrections for file-type comparisons, statement distributions, and repository-level presence.

A.1 Method details

  • The existing-taxonomy comparison corrects the model-labeled share using sensitivity and specificity measured on 324 reference files, clipping estimates to [0, 1].The correction follows Rogan and Gladen (1978).Source: §A.1 · p. 14
  • For statement-level distributions, the method maps Haiku’s code distribution to Sonnet’s using a confusion matrix estimated on their overlap.Source: §A.1 · p. 14
  • For repository-level code presence, it calibrates Haiku’s observed share using Sonnet’s conditional detection probabilities, applying calibration only when it lowers held-out error.The calibrated share is p1q + p0(1 − q).Source: §A.1 · p. 15

A.1.1 Population and census

The census defines adopters through exact-name root instruction files in a popular, active, language-bounded GitHub frame. It also records file relationships, history, and other instruction formats, with limits on what those snapshots capture.

A.1.1 Population and census

  • The study checked all 22,854 repositories returned by GitHub’s search API against popularity, recent-activity, non-fork, non-archived, and thirteen-language criteria.The frame reflects GitHub’s search index; a qualifying push may be on any branch and may be by a bot.Source: §A.1.1 · p. 15
  • An adopter is a repository with an exact-name root AGENTS.md or CLAUDE.md on the default branch; this measures file presence, not agent use.Nested files, case variants, and other instruction formats are excluded from that definition.Source: §A.1.1 · p. 15
  • The census is not conditioned on prior agent use, unlike studies drawing on repositories already associated with coding agents.Source: §A.1.1 · p. 15
  • Files were read at the adopter’s snapshot commit, and repository metadata included activity and AI-marked commits among the latest 100 default-branch commits.AI-marked commits are a lower bound on AI-assisted commits because trailers or signatures must be added and projects may forbid them.Source: §A.1.1 · p. 15
  • The study classified how AGENTS.md and CLAUDE.md relate, including links, imports, pointers, added content, deference, and identical copies.Changing the character threshold from 100 to 2,000 did not change the deference ratio, though it changed the count of repositories with two files containing their own content.Source: §A.1.1 · p. 15
  • History analysis covered 8,393 files with resolvable path histories, but only surviving snapshot files and renames are not followed.Consequently, time trends describe surviving files, and a CLAUDE.md renamed to AGENTS.md counts as AGENTS.md-first; other formats were counted separately, with non-adopters checked at a later HEAD.Source: §A.1.1 · p. 16

A.1.2 Units of analysis

This section describes how the study defines and interprets repository instruction sets, combining model-assisted coding with human review while noting limits of automated interpretation. It also documents the collaboration-rule taxonomies and deviations from planned analyses.

A.1.3 Structure and wording

  • Document features captured Markdown structure and prose wording, but detector counts were lexical rather than semantic classifications.Model-assisted checks in context led to correction or removal of three detectors.Source: §A.1.3 · p. 16

A.1.4 The prior codebook

  • The prior 16-type codebook was applied to 6,578 files, with section labels combined into file-level labels and prevalence corrected for the model’s measured sensitivity and specificity.Prompt calibration used 324 reference-study files split into revision and reporting halves; the model assigned more types than the human coders.Source: §A.1.4 · p. 16

A.1.5 Statements, subjects and elements

  • The statement codebook has 10 dimensions and 48 values, including 14 statement-function values; segmentation produced 341,852 statements across 6,577 files.In 99.4% of statements, spans appeared verbatim in the source, which checks copying but not segmentation accuracy; Haiku left some statements out in 75 files, which were excluded.Source: §A.1.5 · pp. 17–18

A.1.6 Prespecified hypothesis tests

  • Prespecified repository-level tests used 5,587 adopters, regression models, and Benjamini–Hochberg correction across 11 tests at q = 0.05.Outcomes came from uncorrected Haiku codes, so annotation error could bias associations; robustness checks used Sonnet codes, an alternative AI measure, and exclusions of the two largest languages.Source: §A.1.6 · pp. 18–19

A.1.7 Exploratory associations with project characteristics

  • Exploratory content models used 5,586 instruction sets and controlled the false discovery rate within a 99-test family; adopter-control comparisons were stratified by star tier and language.The analyses were planned before results were computed and were framed as searches for associations, not tests of directional hypotheses.Source: §A.1.7 · p. 19

A.2 Human review and agreement with model annotations

  • 92% of reviewed statement boundaries, 93% of restatements and extracted elements, and 95% of free-text subjects were accepted; blind human-model agreement was lower on categories and subjects.The models agreed with each other more than with the reviewer, and the checks support extraction in the reviewed sample—not the later model-defined subject and element assignments.Source: §A.2 · p. 20, Table 3 · p. 20

A.5 The two file names

  • CLAUDE.md deferred to AGENTS.md in 1,540 repositories, versus 182 in the reverse direction, and the roughly eight-to-one ratio held across thin-import thresholds.Among repositories that started with CLAUDE.md, 396 later turned it into a bridge to AGENTS.md; 66.9% of conversions occurred when AGENTS.md was created.Source: §A.5 · p. 21, Table 6 · p. 22

A.6 Origins per instruction set

  • Origin traces were measured across 5,998 instruction sets and file histories, using generator signatures, tool-inserted blocks, shared text, and near-duplicates.These traces identify signatures, shared text, or marked commits; they do not establish authorship.Source: §A.6 · p. 22, Table 8 · p. 23

A.10 Element taxonomies

  • The four collaboration subjects—pull-request descriptions, commit messages, test writing, and code review—are tabulated by whether rules require or prohibit each element.The tables cover 1,345, 1,739, 2,586, and 401 files with a rule on the respective subject; definitions and examples are released with the data.Source: Table 17 · p. 26, Table 18 · p. 26, §A.10 · p. 26

A.11 Deviations from the analysis plan

  • The planned mixed logistic model was replaced by a repository-clustered GEE because the mixed model was unstable, and the paired test population was corrected before reporting.Exploratory analyses also differed from the prespecified list: file length used stars only, composition was modeled as a main effect rather than an interaction, and per-language subject profiles were not run. Post hoc checks added an AI-share analysis excluding 141 repositories whose files prohibit AI attribution, single-predictor collaboration models, and a paired effect in percentage points with a bootstrap interval.Source: §A.11 · p. 28

Related papers

Loading 2610.31715v2…