Source-linked AI summary
On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents
Jai Lal Lulla, Seyedmoein Mohsenimofidi, Matthias Galster, Jie M. Zhang, Sebastian Baltes, Christoph Treude
TL;DR
Repository-level instructions may affect the operational efficiency of increasingly autonomous AI coding agents, but empirical evidence has been limited. This study compares Codex executions with and without AGENTS.md on real pull-request tasks, measuring runtime and token use. AGENTS.md is associated with lower median completion time and output-token consumption, while the study identifies correctness and alignment as directions for future evaluation.
Problem
The study addresses limited empirical evidence about how repository-level AGENTS.md instructions affect AI coding-agent efficiency.
Method
The study compares OpenAI Codex executions with and without a root AGENTS.md file on real GitHub pull requests, measuring wall-clock time and token usage.
Results
Median completion time decreases by ≈28.64%, while median output token usage decreases by ≈16.58% with AGENTS.md.
Takeaways & Limitations
Repository-level instruction files have measurable operational associations with faster task completion and lower token usage in this study.
Takeaways & Limitations
The study’s efficiency metrics do not establish whether agent-produced changes are correct, maintainable, or aligned with developer intent.
Abstract
from arXiv · showhide
AI coding agents such as Codex and Claude Code are increasingly used to autonomously contribute to software repositories. However, little is known about how repository-level configuration artifacts affect operational efficiency of the agents. In this paper, we study the impact of AGENTS$.$md files on the runtime and token consumption of AI coding agents operating on GitHub pull requests. We analyze 10 repositories and 124 pull requests, executing agents under two conditions: with and without an AGENTS$.$md file. We measure wall-clock execution time and token usage during agent execution. Our results show that the presence of AGENTS$.$md is associated with a lower median runtime ($Δ28.64$%) and reduced output token consumption ($Δ16.58$%), while maintaining a comparable task completion behavior. Based on these results, we discuss immediate implications for the configuration and deployment of AI coding agents in practice, and outline a broader research agenda on the role of repository-level instructions in shaping the behavior, efficiency, and integration of AI coding agents in software development workflows.
1 Introduction
AI coding agents increasingly perform multi-step repository tasks, but the effects of persistent repository-level instructions remain insufficiently understood. This study examines AGENTS.md as an operational factor in agent behavior and efficiency.
- AI coding agents now support substantial software-development activities, including code generation, testing, repair, review, and documentation.
- Repository-level files such as AGENTS.md encode project-specific architecture, commands, conventions, and constraints for agents.
- Prior studies characterize agent context files, but their concrete effects on AI coding-agent behavior remain largely unexplored.
- Understanding these effects matters because repository-level instructions may influence operational implications related to cost, scalability, and workflow integration.
- The study compares executions with and without AGENTS.md using wall-clock time and token consumption as measurable efficiency indicators.
2 Background and Related Work
Repository-maintained instruction files are increasingly used to guide AI coding agents, yet evidence isolating their effects on agent efficiency is limited. Prior work has examined related context artifacts and autonomous coding-agent benchmarks without resolving this gap.
- AGENTS.md and similar files are emerging across agent tooling ecosystems as repository-level mechanisms for scoping and guiding agent behavior.
- No prior empirical study identified here isolates AGENTS.md’s effect on token usage or wall-clock completion time while holding tasks, repositories, and agent architecture constant.
- Benchmarks such as SWE-bench support controlled evaluation of autonomous coding agents using real GitHub issues, repository snapshots, and task outcomes.
3 Study Design
The study asks whether a root AGENTS.md file reduces the resources required by an autonomous coding agent to complete repository development tasks. It operationalizes resources as token usage and wall-clock completion time in paired comparisons.
- The study models a developer workflow in which an agent receives a GitHub-like task, modifies a repository, and produces an intended code change.
- The research question tests whether a repository-root AGENTS.md file reduces resources required to complete a development task.
- Resources are measured as token usage and wall-clock time-to-completion during each task run.
- The comparison uses paired runs with and without AGENTS.md while keeping the task and repository snapshot constant.
3.1 Data Collection and Analysis
The study uses OpenAI Codex to evaluate qualifying root AGENTS.md files across sampled repositories and small, repeatable merged pull-request tasks. Data collection filters repository configurations and standardizes task descriptions for comparison.
- OpenAI Codex is used consistently because it supports repository-scale context and tool use for software-engineering tasks.
- A lightweight Python wrapper interfaces with the Codex CLI and automates task setup and metric collection.
- The study focuses on repositories containing one AGENTS.md file at the root to minimize confounding from multiple instruction-file configurations.
- Content filtering retains files providing actionable context such as coding conventions, architecture, project descriptions, and testing guidance.
- An LLM-based classification followed by manual verification retains 26 repositories with qualifying root AGENTS.md files.
- The evaluation randomly samples 10 repositories and selects up to 15 small-scope merged pull requests per repository under size and file-count constraints.
3.1.3 Pull Request Selection and Task Construction.
The study restricts its initial sample to tractable, code-changing pull requests and reconstructs each task from the repository state immediately before merging.
- Pull Request Selection and Task Construction.: Size, scope, and change-type constraints reduce variance from large refactorings and keep agent runs tractable and repeatable.The study excludes documentation-only updates and version bumps, while broader changes remain part of the longer-term research agenda.
- Pull Request Selection and Task Construction.: Each task is grounded in the repository state immediately before the selected pull request was merged.The agent is tasked with recreating the pull request’s changes from that pre-merge state.
3.1.4 Reconstructing Pre-PR Repository State.
The researchers reconstruct historically faithful pre-merge repositories and standardize task input by generating GitHub-issue-style descriptions when pull-request metadata is insufficient.
- Reconstructing Pre-PR Repository State.: The repository is checked out at the pre-merge commit, and the AGENTS.md version present at that commit is extracted.This preserves the repository contents and instruction file that would have been available before the pull request was merged.
- Reconstructing Pre-PR Repository State.: Many pull requests lack sufficient natural-language context, such as an empty body, vague title, or missing linked issue.The researchers address this by generating a consistent task statement for each pull request.
- Reconstructing Pre-PR Repository State.: A local LLM generates each task statement from the pull-request patch and the pre-merge repository structure.The resulting description resembles a GitHub issue and includes expected behavior, constraints, and acceptance criteria.
- Reconstructing Pre-PR Repository State.: The agent is run on each task instance under two conditions.This establishes the paired execution setup used for the comparison.
3.1.6 Experimental Conditions. •
The experiment varies only whether the repository snapshot contains its root AGENTS.md file, while holding the task, repository state, and other files constant across paired runs.
- Experimental Conditions.: With AGENTS.md, the pre-merge repository snapshot includes the extracted root file; without it, the same snapshot has that file removed.All other repository files remain unchanged between conditions.
- Experimental Conditions.: Both conditions provide the agent with the same GitHub-like issue generated for the corresponding pull request.The agent then produces code changes for the original pull-request task.
- Experimental Conditions.: The paired within-task design controls for repository, task, and codebase state while varying only AGENTS.md presence.This isolates the repository-level instruction-file comparison within each task.
- Experimental Conditions.: Experiments use isolated Docker environments, with a fresh repository clone and cleaned workspace for each task.The agent is restricted to modifying files inside the sandboxed container.
- Experimental Conditions.: The study collects operational performance metrics during each run.The listed metrics are introduced as the experiment’s operational measurements.
3.1.8 Metrics. •
The study measures token usage and wall-clock completion time, while using a manual sanity check rather than a full evaluation of output quality.
- Metrics.: Token usage includes input tokens, cached input tokens, and output tokens.This metric aggregates the token categories consumed during an agent run.
- Metrics.: Time-to-completion is the wall-clock time, in seconds, required for the agent to produce its final output.It captures the run’s elapsed operational duration.
- Metrics.: A comprehensive evaluation of semantic correctness or functional equivalence to the merged pull request is outside the paper’s scope.The authors identify such evaluation as part of their research roadmap.
- Metrics.: Table 1 presents resource usage with and without AGENTS.md.The table is framed as a comparison of resource usage across the two conditions.
- Metrics.: The manual sanity check verifies that runs were aimed at the intended task rather than aborted runs or random edits.It provides confidence that efficiency measurements were not driven by obvious failures, but it is not a full correctness evaluation.
4 Results
AGENTS.md is associated with faster task completion and lower token consumption in agent runs. The runtime reduction appears broadly distributed, while the token reduction is concentrated in a small number of high-cost runs.
- Token usage: 16.58%: Median output token usage decreases from 2,925.00 tokens without AGENTS.md to 2,440.00 with it.Mean output token usage decreases by approximately 20.08%, from 5,744.81 to 4,591.46 tokens.
- Token usage: The larger reduction in mean than median output tokens suggests savings are concentrated in a small number of very high-cost runs.The result does not indicate a uniform reduction across all task instances.
- Wall-clock time-to-completion: 28.64%: Median wall-clock time-to-completion decreases from 98.57s without AGENTS.md to 70.34s with it.Mean completion time also decreases by 20.27%, from 162.94s to 129.91s.
- Wall-clock time-to-completion: The close alignment between mean and median runtime improvements indicates faster completion is not driven solely by extreme runs.Both measures shift toward shorter task completion times with AGENTS.md.
5 Research Roadmap
The paper presents AGENTS.md as a practical repository-level mechanism with measurable operational effects, while framing the evidence as an initial step. Its roadmap calls for broader validation, richer behavioral evaluation, and investigation of how file properties and execution traces explain efficiency effects.
- Current evidence: The study provides early evidence that AGENTS.md can affect agent efficiency in realistic development tasks.The authors describe the study as an initial step toward understanding repository-level instruction effects.
- Broader empirical scope: Future studies should replicate the evaluation across more repositories, diverse pull requests, agent systems, and model families.The roadmap also proposes relaxing task-size and scope constraints to test larger refactorings and more complex scenarios.
- Beyond efficiency: Efficiency metrics alone do not capture whether agent-produced changes are correct, maintainable, or aligned with developer intent.The authors propose adding correctness and alignment evaluations beyond token usage and wall-clock time.
- Mechanisms and file properties: Execution-trace analysis may clarify whether efficiency gains reflect fewer planning iterations, less exploratory navigation, or fewer repeated model requests.The authors also plan to study file specificity, organization, and workflow guidance rather than treating AGENTS.md as only present or absent.
- Implications: The summary positions repository-level instruction files as a practical mechanism for shaping agent behavior and motivating research on efficiency, alignment, and workflow integration.The conclusion describes the reported effects as initial empirical evidence from real pull requests.