Source-linked AI summary
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, Martin Vechev
TL;DR
Context files are widely recommended for adapting coding agents to repositories, but their impact on real-world task resolution had not been rigorously established. The paper evaluates agents with no context file, LLM-generated files, and developer-committed files on SWE-BENCH and CTXBENCH, finding no general task-success improvement despite higher cost. It concludes that context files should focus on specific additional instructions and be evaluated before adoption.
Problem
The effectiveness of widely adopted context files for coding agents on real-world software-engineering tasks had not been rigorously studied.
Method
The paper evaluates coding agents across SWE-BENCH and CTXBENCH using no context file, LLM-generated files, and developer-committed files.
Results
Context files do not significantly improve task success rates, while increasing cost and the number of steps required to complete tasks.
Takeaways & Limitations
Context files should contain specific instructions beyond the codebase and be rigorously evaluated before adoption.
Takeaways & Limitations
The evaluation focuses on Python, so context-file effects may differ for less training-represented programming languages and toolchains.
Abstract
from arXiv · showhide
A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files, and a novel collection of issues from repositories containing developer-committed context files. Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files. Specifically, we find that while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful. We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.
1 Introduction
Context files are widely adopted to help coding agents navigate repositories and follow local practices, but their effectiveness on real-world tasks had not been rigorously studied. This work introduces CTXBENCH and finds that context files generally do not improve performance while increasing cost.
- Context files such as AGENTS.md are widely supported and included in over 60,000 open-source repositories.
- Their intended purpose is to help agents navigate repositories, run tools correctly, follow conventions, and solve tasks more often.
- CTXBENCH contains 138 Python bug-fixing and feature-addition instances from 12 repositories with developer-committed context files.
- The evaluation compares no context file, an LLM-generated context file, and a developer-committed context file across CTXBENCH and SWE-BENCH.
- Context files do not significantly improve performance, although agents follow their instructions and perform more exploration, testing, and reasoning, increasing costs by over 20%.
2 Background and Related Work
Coding agents use tools to autonomously resolve repository-level tasks, and developers increasingly provide context files to guide them. Prior work mainly described context-file content, leaving their effect on agent behavior and task performance untested at scale.
- Background: Coding agents are LLM-based systems whose harnesses let them execute commands, search the web, and read or modify files.
- Background: Strong performance on repository-level benchmarks has accelerated coding-agent adoption and development by specialized companies and model providers.
- Context files: Model and agent developers recommend context files such as AGENTS.md and CLAUDE.md to provide repositories with additional guidance.
- Evaluating context files: Prior studies categorized context-file content and reported descriptive metrics, but did not investigate effectiveness.
- Repository-level evaluation: This work evaluates whether actively used context files improve autonomous issue-resolution and feature-addition capabilities.
3 CTXBENCH
CTXBENCH is a SWE-BENCH-like benchmark built from real pull requests in Python repositories containing developer-written context files. Its instances pair tasks and repository states with validated tests for measuring coding-agent task resolution.
- 3.1 Notation and Definitions: CTXBENCH targets the evaluation of developer-provided context files and defines instances using issues, repositories, test suites, and golden patches.
- Finding repositories: Candidate repositories were required to contain a root context file, use Python, include tests, and have at least 400 publicly documented pull requests.
- Filtering pull requests: Pull requests had to reference an issue, modify Python code, and describe deterministic, testable behavior; unit-test modifications were not required for niche repositories.
- Task Descriptions and Tests: The construction process standardized task descriptions, generated unit tests when needed, and manually reviewed tests for overfitting.
- Evaluation: Evaluation prompts agents with each task and measures whether the predicted patch makes the selected tests pass.
- Overview of CTXBENCH: 138 instances were obtained from 5,694 pull requests across 12 repositories, with repository distributions visualized in Figure 2.
4 Experimental Evaluation
The evaluation compares coding agents across SWE-BENCH and CTXBENCH under no-context, LLM-generated, and developer-provided context settings. Context files generally fail to improve resolution while increasing steps, inference cost, exploration, and testing, with effects varying across context sources and ablations.
- Experimental setup: The study evaluates four coding agents on SWE-BENCH Lite’s 300 tasks and CTXBENCH’s 138 instances using three context-file settings.The settings are NONE, LLM-generated, and developer-provided context files; developer-provided files are available only for CTXBENCH.
- Main results: LLM-generated context files reduce average resolution by 0.5% on SWE-BENCH and 2% on CTXBENCH, without significant performance effects.They reduce performance in 5 of 8 settings, with reported p-values of 87% and 37%.
- Main results: 20% and 23% average cost increases accompany LLM-generated context files on SWE-BENCH and CTXBENCH, respectively, as steps rise in every setting.Average step increases are 2.45 and 3.92, respectively, with a cost-increase p-value below 0.001%.
- Main results: Developer-provided context files improve performance by 2.4% on average and significantly outperform LLM-generated files, while increasing steps by 3.34 and cost by at most 19%.They improve performance for every agent except CLAUDE CODE.
- Behavioral changes: Context files are followed well and increase testing, repository navigation, repository-specific tooling, and reasoning, but do not provide effective repository overviews.Agents use mentioned tools substantially more often, including uv at 1.6 versus fewer than 0.01 uses per instance and repository-specific tools at 2.5 versus fewer than 0.05.
- Behavioral changes: Context-file overviews do not meaningfully reduce the steps before agents first interact with files modified in the original patch, and can prompt repeated searches and reading of context files.The effect holds across SWE-BENCH, CTXBENCH, and all agents, while increasing steps significantly for GPT-5.1 MINI.
- Ablations: Stronger generating models improve SWE-BENCH performance by 2% on average but worsen CTXBENCH performance by 3%, showing that stronger models do not necessarily produce superior context files.The comparison uses context files generated by GPT-5.2 with CODEX and SONNET-4.5 with CLAUDE CODE versus self-generated files.
- Ablations: Prompt choice does not consistently determine performance: the CODEX prompt helps some settings and hurts others, while neither agent-matched nor specific prompts consistently perform best.The overall sensitivity to prompt choice is described as small.
5 Limitations and Future Work
The evaluation is limited to Python and task-resolution performance, leaving effects on other languages, toolchains, efficiency, and security for future work. The authors also identify improving automatic context-file generation as an open direction.
- Programming languages: The evaluation focuses on Python, whose strong representation in training data may reduce context files’ observable effect.Future work may examine less common programming languages and toolchains.
- Context files beyond task resolution: The study measures task resolution rate, while code efficiency and security remain unevaluated aspects of coding-agent performance.The authors identify both as directions for future work.
- Improving context file generation: Improving the automatic generation of useful context files is an open research direction.The authors position the work as a baseline for evaluating automatically generated context files.
6 Conclusion
Across SWE-BENCH and CTXBENCH, context files do not significantly improve task success, although their instructions are generally followed and increase testing and exploration. The results support using context files for specific additional instructions rather than repository overviews.
- Results: Context files consistently increase cost and steps, while LLM-generated files marginally reduce success and developer-written files marginally improve it, neither significantly.The evaluation covers four coding agents on SWE-BENCH and CTXBENCH.
- Trace analysis: Instructions in context files are generally followed and lead to more testing and broader exploration, but repository overviews are ineffective.The trace analysis distinguishes compliance with instructions from the usefulness of repository summaries.
- Conclusion: Context files should contain specific additional instructions beyond information already available in the codebase.This is the paper’s supported practical conclusion about context-file content.
A Experimental Details
The supplied experimental-details passage only introduces the section and provides no substantive experimental information.
- The section provides additional details about the experiments from §4.
A.1 Additional Experimental Details
This subsection continues the experimental details and specifies the coding environment for CTXBENCH and SWE-BENCH.
- The subsection continues the experimental details for experiments in §4.2.
- Coding environment: CTXBENCH agents run in Docker with basic tooling and Internet access, without git history or remotes; SWE-BENCH uses Jimenez et al.’s provided Docker images.The authors manually checked that agents did not cheat, including by inspecting corresponding pull requests.
A.2 Trace Analysis
Trace analysis finds that context files are followed by coding agents and increase exploration, testing, and code-quality checks, but do not improve repository-level success rates. These additional actions significantly increase steps and cost.
- Instruction following: Mentioning a tool in a context file increases its usage, especially for uv, pytest, and repository-specific tools.This indicates that agents follow explicit tooling instructions.
- Trace effects: Context files significantly increase testing, codebase exploration, and code-quality checks.The analysis aggregates tool calls into high-level intent categories.
- Task performance: Context files do not significantly affect success rates for repositories in SWE-BENCH or CTXBENCH.No individual repository shows a significant impact from context files.
- Cost and steps: Context files systematically and significantly increase the average steps and cost required to solve instances.These effects are assessed with stratified permutation tests.
B Additional Results
Additional analyses show that context files can help when repository documentation is removed, but otherwise their length and common content categories do not explain accuracy improvements. Context files also do not show a clear relationship between length and performance or cost.
- Documentation removal: 2.7% average accuracy improvement occurs when context files are the only documentation available.In this setting, LLM-generated files consistently outperform developer-written files across settings.
- File length: Context-file length shows no clear dependency with success rate or per-instance cost.The analysis suggests instruction following, rather than file length, better explains increased cost.
- Category ablations: No common category in LLM-generated context files significantly improves or worsens benchmark accuracy.The examined categories are Overview, Tooling, and Testing.
- Category ablations: Testing increases cost significantly on both benchmarks, while Tooling increases cost significantly on SWE-BENCH.No category has a significant positive or negative effect on benchmark accuracy.
- Contamination: Splitting CTXBENCH instances by model knowledge cutoffs does not change the finding that context files do not improve performance.The contamination analysis compares no-context, LLM-generated, and human context-file settings.
C Validating CTXBENCH instances
The authors validate CTXBENCH by comparing developer- and LLM-generated context files and inspecting task specifications. CTXBENCH is described as slightly easier but fairer than SWE-BENCH under generated issue descriptions.
- Context-file validation: Developer-provided context files tend to be shorter and less likely to be AI-generated than LLM-generated files.The comparison uses length distributions and Pangram AI-detection scores.
- Instance validation: Most CTXBENCH instances provide clear reproduction instructions and technical issue descriptions.The authors manually inspect all instances and unit tests.
- Instance validation: 25 of 138 CTXBENCH instances receive manual test edits to remove over-specified tests.The edits are intended to improve the specification of expected behavior.
- Benchmark comparison: Generated issue descriptions produce about 15% higher SWE-BENCH accuracy while preserving model rankings.The authors interpret this construction as slightly easier but fairer than the original descriptions.
Datasets
The dataset-generation materials define procedures for selecting deterministic, testable pull requests, constructing issue descriptions and regression tests, and categorizing coding-agent tool calls. They also specify reproducibility and output requirements.
- Dataset construction: Pull requests are filtered for issue references, Python-file modifications, and deterministic, testable behavior suitable for SWE-BENCH-like regression tests.The filtering combines rule-based checks with an LLM agent.
- Issue generation: Issue descriptions require reproducible minimal reproduction steps and must avoid leaking the pull request’s solution.Feature requests additionally require precise specifications independent of implementation details.
- Test generation: Regression tests must be deterministic, target expected behavior, fail before the patch, and pass after it.Tests should avoid incidental implementation details and align exactly with any provided specification.
- Test generation: Generated test workflows require new test files, project-consistent test style, and scripts that report results in JSON.The workflow also specifies commands for running only the created tests.
- Trace analysis: Tool calls are assigned reusable, lowercase verb-plus-object intent labels and mapped into standardized tool categories.The labeling excludes paths, flags, IDs, repository names, and counts from category names.