PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

Prakhar Khatri

arXiv:2607.27250v1cs.SEcs.AI

TL;DR

Persistent context files are widely used, yet prior evidence is contradictory about whether they improve coding-agent outcomes. This paper conducts a controlled ablation across two agents and real repository tasks, finding no measurable correctness shift within bounded margins. It attributes failures to implementation skill rather than missing repository knowledge and shows that task difficulty is agent-specific.

  • Problem

    Persistent context files are widely used, yet prior evidence is contradictory about whether they improve coding-agent outcomes.

    Source: §1 · p. 1

  • Method

    The study conducts a controlled ablation of NONE, ALWAYS ON, and SELECTIVE injection across two agents, real tasks, hidden gold tests, and repeated within-task comparisons.

    Source: §1 · p. 1, §3.3 · p. 3, §3.6 · p. 4

  • Results

    Context-injection strategy does not measurably move correctness, with observed effects bounded to <10–15pp; surviving efficiency effects are narrow and process-level.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Failures reflect implementation skill rather than missing repository knowledge, and authentic AGENTS.md never converts a near-miss failure into a pass.

    Source: §6 · p. 8

    Correctness equivalence bounds remain limited by the small sample: 15/17 tasks with three repeats yield MDE >30pp, and narrower bounds would require approximately 120 tasks.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.

1 Introduction

Repository context files are widely used to guide coding agents, but prior studies disagree about whether they improve outcomes. This paper reconciles that conflict with a controlled two-agent ablation, finding small bounded correctness effects and failures driven by implementation skill rather than missing repository knowledge.

Motivation

  • Prior studies report contradictory effects: AGENTS.md improves efficiency in one study but does not significantly affect task completion in another.The studies differ in agent family, evaluation method, and experimental control.Source: §1 · p. 1

Study design

  • A controlled ablation varies NONE, ALWAYS ON, and SELECTIVE context injection across frontier agents using real pull requests and hidden gold tests.The strategies primarily differ in how repository context is delivered.Source: §1 · p. 1
  • 288 evaluated runs bound correctness effects to ≤10pp for Claude and ≤15pp for Codex.The design includes 15 Claude tasks and 17 Codex tasks, each tested under three strategies with three repeats.Source: §1 · p. 1

Findings

  • Failure-mode triage attributes task failures to implementation skill rather than missing repository knowledge, while the real AGENTS.md never turns a near-miss into a pass.The manipulation probe directly tests whether authentic repository guidance rescues borderline failures.Source: §1 · p. 1
  • Borderline task difficulty is agent-specific, suggesting that single-agent studies can sample different informative difficulty bands and reach contradictory conclusions.The paper presents this as a candidate explanation for prior disagreements.Source: §1 · p. 1

2 Related Work

Prior work studies coding-agent evaluation, memory, retrieval, and context arrangement, but does not isolate static repository context across agent families. This paper positions injection strategy as distinct from retrieval and harness mechanics and uses equivalence testing to characterize null effects.

Coding agents and benchmarks

  • SWE-bench established issue-to-PR evaluation, while agent–computer-interface research shows harness design can affect performance independently of the underlying model.This study instead examines persistent repository context.Source: §2 · p. 2

Context and memory for agents

  • Persistent memory and long-context studies motivate whether static AGENTS.md guidance is actually attended to when loaded at session start.Prior work reports uneven use of information by position in long prompts.Source: §2 · p. 2
  • Earlier AGENTS.md studies report efficiency gains for Codex-family agents or no correctness effect for Claude-family agents, but neither controls injection strategy across agent families.The paper identifies agent, evaluation, and control differences as unresolved confounds.Source: §2 · p. 2

Repository-level context for code generation

  • Repository-level retrieval methods retrieve code or documentation at query time, whereas AGENTS.md is a static author-curated context layer studied here as an orthogonal variable.The selective strategy uses topic-organized repository material rather than ordinary query-time retrieval alone.Source: §2 · p. 2

Equivalence testing in empirical SE

  • The study adopts TOST alongside permutation tests and power analysis because conventional null-hypothesis testing cannot confirm absence of an effect.TOST provides bounded equivalence claims under the specified margin.Source: §2 · p. 2

3 Method

The method pairs three context-injection strategies with real repository tasks, two frontier agents, hidden gold-test evaluation, and task-level statistical analyses. The design controls workspace and evaluation conditions while documenting important corpus, channel, metric, and sample-size limitations.

3.1 Overview

  • The within-task design runs each task under NONE, ALWAYS ON, and SELECTIVE with three repeats across two agents, producing 291 completed runs and 288 evaluated correctness cells.Efficiency metrics use all completed runs, whereas three failed Claude evaluations are excluded from correctness.Source: §3.1 · p. 2

3.2 Task Selection and Evaluation

  • Tasks come from merged pull requests in three Python repositories, use PR tests as hidden gold tests, and include 17 Codex tasks plus 15 Claude tasks after borderline screening.The screening sweep targeted tasks with intermediate pass rates and exposed an effort-classifier portability bug.Source: §3.2 · p. 3

3.3 Independent Variable: Injection Strategy

  • NONE removes AGENTS.md, ALWAYS ON injects the full file every turn, and SELECTIVE enables on-demand retrieval of topic-organized wiki files.In every condition the original file is removed from the workspace; SELECTIVE is content-matched only for opshin and uses broader wikis for pdm and firebase.Source: §3.3 · pp. 3–4

3.4 Agents

  • Claude Code and Codex use the same workspace, task prompt, and evaluation pipeline, but context enters through different channels: system prompt for Claude and user-prompt prepend for Codex.This injection-channel asymmetry is a stated confound.Source: §3.4 · p. 4

3.5 Efficiency Metrics

  • Tool calls, duration, and output tokens are portable efficiency metrics, while cache tokens are Claude-specific; turns are excluded because Codex emits one turn event regardless of work performed.The turn metric failure shows why cross-agent metrics require portability checks.Source: §3.5 · p. 4

3.6 Statistical Analysis

  • Analyses average repeats within each task, then use within-task permutation tests for correctness, paired Holm-corrected Wilcoxon tests for efficiency, clustered-bootstrap TOST, and Monte Carlo power analysis.Table 1 reports pass rates by strategy and agent, with omnibus permutation tests assessing whether strategy explains correctness beyond task difficulty.Source: §3.6 · p. 4, Table 1 · p. 4

4 Results

Across both agents, context-injection strategy does not measurably change correctness, while producing narrow, agent- and repository-specific process effects. Borderline difficulty is agent-specific, classifier metrics are not portable, and task-level variance limits detection power.

4.1 Correctness: No Strategy Effect (Both Agents)

  • Neither agent shows a statistically significant correctness effect from NONE, ALWAYS ON, or SELECTIVE context injection.Pairwise differences are at most 2.3pp for Claude and 5.9pp for Codex, with omnibus permutation tests of p=1.00 and p=0.66.Source: §4.1 · p. 4
  • 58% vs. 42% vs. 42% pass rates on four Codex-borderline tasks show no context benefit where the design has dynamic range.The comparison is NONE vs. ALWAYS ON vs. SELECTIVE.Source: §4.1 · p. 5
  • <10pp for Claude and <15pp for Codex bound every pairwise strategy difference, although the small task sample does not provide a powered equivalence claim.The bounds come from descriptive TOST on a task-clustered bootstrap and remain compatible with a >30pp minimum detectable effect.Source: §4.1 · p. 5

4.2 Efficiency: Two Narrow Process Signals, Otherwise Null

  • Claude’s context strategies reduce cache-creation tokens, but Codex efficiency metrics remain flat across strategies.Claude’s cache-creation reduction is significant, while turns and tool calls show no corrected significant effect; Codex tool calls, output tokens, and duration are nearly unchanged.Source: §4.2 · p. 5
  • ∼24% lower wall-clock time on Claude’s opshin tasks coincides with blind full-suite runs falling NONE 3.67 → ALWAYS ON 2.44 → SELECTIVE 1.67.This exploratory process effect is narrow: it appears on opshin for Claude, not as a correctness improvement or a general cross-agent effect.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 Agent-Specific Borderline Tasks

  • Spearman ρ=0.75 shows that task difficulty transfers imperfectly across agents, with roughly 40% of shared tasks occupying different informative bands.Six shared tasks are borderline for exactly one agent, and six differ in floor/ceiling status.Source: §4.3 · p. 6, Table 3 · p. 6

4.4 Classifier Portability Lesson

  • Different agent architectures make turn-based effort screening non-portable: Codex emitted one turn.completed event per session, dropping eight genuinely high-effort tasks.Those tasks involved 18–51 tool calls and 410–1220 seconds; tool-call screening recovered them.Source: §4.4 · p. 6

4.5 Power Analysis and Detection Limits

  • At n=17 and three repeats, even a 30pp effect is detected only 57% of the time, so scaling requires more tasks rather than more repeats.Task-level variance dominates the power calculation.Source: §4.5 · pp. 6–7

5 Discussion

The null result reflects implementation-skill failures rather than missing repository knowledge, and a manipulation probe shows that even high-quality real AGENTS.md files do not rescue near-miss tasks. The findings suggest prior contradictions may reflect agent-specific task difficulty, while context can still affect process efficiency without changing correctness.

5.1 Why the Null Holds: Skill, Not Knowledge

  • Implementation-skill gaps—not missing repository knowledge—explain near-miss failures, including precision bugs, wrong architectural patterns, miswired checks, and deep type-system reasoning.The triage covers optimization correctness, proactive token refresh, validator-argument rejection, and type narrowing.Source: §5.1 · p. 7

5.2 Manipulation-Validity Probe

  • The real AGENTS.md never converts a near-miss failure to a pass on either agent in the manipulation-validity probe.Table 4 compares NONE, ALWAYS ON, and SELECTIVE across two convention-closest near-miss tasks; the probe therefore tests more than a bare null result.Source: Table 4 · p. 7, §5.2 · p. 7

5.3 Reconciling Prior Work

  • Context files do not improve correctness on either agent family, while reported efficiency differences may arise from injection mechanics rather than capability.The discussion attributes prior efficiency gains to prompt-size costs and notes that SELECTIVE achieves a cache-efficiency reduction without improving task performance.Source: §5.3 · pp. 7–8
  • Prior disagreements may reflect agent-specific borderline difficulty, with tasks informative for one agent landing in another agent’s floor or ceiling.The paper presents this reconciliation as a hypothesis because the prior studies’ exact task sets were unavailable.Source: §5.3 · pp. 7–8

5.4 Practitioner Implications

  • In three Python repositories, generic context files show no correctness benefit but may reduce wasted full-suite test runs and remain useful on cost and latency grounds.The authors caution that benefits for other languages, larger repositories, or purpose-built task-specific context remain unresolved.Source: §5.4 · p. 8

6 Conclusion

The study finds that context-injection strategy does not measurably change coding-agent correctness, while its efficiency effects are narrow and process-level. Failure analysis points to implementation skill as the gating factor, and the authors release resources for larger, better-powered studies.

6 Conclusion

  • Context-injection strategy does not measurably move correctness across two frontier agents and 17 real-world tasks, with observed effects bounded to <10–15pp.The descriptive TOST result is not a powered equivalence claim.Source: §6 · p. 8
  • The surviving efficiency effects are narrow and process-level: reduced cache footprint for Claude’s SELECTIVE strategy and fewer blind full-suite test runs in one repository.Neither efficiency effect changes the correctness outcome.Source: §6 · p. 8
  • Failure analysis indicates that real tasks fail on implementation skill rather than missing repository knowledge, and the real context file never converts a near-miss to a pass.A single Claude task showed a non-positive trend at n=3 repeats, which the authors explicitly treat as a single-task observation rather than a general downward effect.Source: §6 · p. 8
  • The authors release the full harness, 291-run dataset, and power-analysis code to support scaling toward approximately 120 tasks for powered equivalence at 10pp.The released materials are intended to support future studies of this design.Source: §6 · p. 8

Limitations

The study’s limitations constrain the precision, generalizability, and interpretation of its context-strategy null result. Key boundaries include small task counts, limited repository diversity, agent-arm asymmetry, non-natural injection, content-mismatched SELECTIVE context, naturalistic files, model snapshots, and mixed run provenance.

Sample size

  • Across 15/17 tasks with 3 repeats, MDE >30pp; a 10pp effect is undetectable; TOST bounds the effect to <10–15pp but cannot achieve narrower equivalence without ∼120 tasks.Source: Limitations · p. 8

Repository diversity

  • Results come from three Python repositories and may not generalize to other languages, larger codebases, or repositories with exceptionally detailed context files.Source: Limitations · p. 8

Injection-channel asymmetry

  • Claude received context through the system prompt while Codex received a user-turn prepend, confounding comparisons between agent arms though not within-agent strategy contrasts.Source: Limitations · p. 8

Ecological validity

  • ALWAYS ON injects context every turn, stronger than natural workspace discovery, so the claim that natural discovery would also fail remains an inference requiring a natural condition.Source: Limitations · p. 8

Selective corpus

  • SELECTIVE uses a content-matched wiki only for opshin but substantially larger auto-generated wikis for pdm and firebase, confounding delivery channel with corpus.This prevents clean attribution of SELECTIVE’s cache-footprint reduction to channel alone.Source: Limitations · p. 9

Manipulation validity

  • Naturalistic context files were rated Good/Excellent, but purpose-built task-specific context could still help and remains untested.Source: Limitations · p. 9

Model-version snapshot

  • Results are specific to claude-sonnet-4-6 and gpt-5.5 at study time, so model updates may change agent behavior and the null.Source: Limitations · p. 9

Mixed provenance

  • Dropping Claude’s local repeat-0 leaves marginal pass-rates at 53–55% and paired contrasts within ±3.3pp, indicating provenance did not drive the correctness null.Source: Limitations · p. 9

Ethics Statement

The study uses public repositories and sandboxed agents without human subjects or personal data. External-state modifications were blocked, and authors verified the AI-assisted drafting and experimental claims.

Scope

  • The study involves no human subjects or personal data; its subjects are AI coding agents operating on public open-source repositories.Source: Ethics Statement · p. 9

Safety

  • An egress-locked sandbox prevents pushes, commits, or pull requests from reaching GitHub.Source: Ethics Statement · p. 9

AI assistance

  • Authors used an AI assistant for drafting and scaffolding, while verifying all experimental results, statistics, and claims against released data.Source: Ethics Statement · p. 9

A Experimental Harness Details

The harness prevents agents from altering external state and evaluates changes against restored gold tests. This isolates task correctness from repository-side effects.

Safety layers

  • Defense-in-depth blocks external modifications through DNS blackholing, remote removal, command shims, token stripping, and pruning future git history.Source: A Experimental Harness Details · p. 10

Evaluation pipeline

  • After completion, the pipeline captures the diff, restores gold tests, applies the gold patch, runs configured pytest, and passes only with zero failures or errors.Source: A Experimental Harness Details · p. 10

B Per-Task Results

Per-task results expose heterogeneous effects rather than a uniform context advantage. In opshin, context reduced full-suite test invocations on three of four tasks, while one task reversed the trend.

Per-task pass-rates

  • Figure 2 reports per-task pass-rates for both agents, with complete pass/fail counts available in the supplementary results summary and database.Source: B Per-Task Results · p. 10

Opshin full-suite runs

  • Three of four opshin tasks used fewer full-suite invocations under context, while task 605 used slightly more: 2.50/0.75, 1.20/1.25, 2.50/0.67, and 9.33/6.25 for NONE/mean-context.Source: B Per-Task Results · p. 10

C Reproduction and Data Availability

The authors release the experimental resources and public repository needed to reproduce the study, including implementations, datasets, analysis code, and per-cell results.

C Reproduction and Data Availability

Loading 2607.27250v1…