Source-linked AI summary
The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks
Bardia Mohammadi, Lars Klein, Aman Chadha, Akhil Arora, Laurent Bindschaedler
TL;DR
Repository-scale coding agents must maintain consistency among coupled facts spread across a repository. The paper tests whether context and model memory supply those facts, finding that availability—not distance or context volume—determines success, while conflicting sources and workload scope bound the account.
Problem
Repository-scale edits require agents to reconstruct cross-file dependencies, but evaluations provide limited evidence about whether the necessary coupled facts are available when writing.
Method
The paper models coherence as a coupled-fact graph and tests fact availability by controlled channel withholding, fault injection, and matched repository-scale workloads.
Results
299 of 300 matched trials reached at least 9 of 12 requirements when the same facts were supplied in the prompt, while withholding facts cost only their supported work.
Takeaways & Limitations
Harnesses should track the facts available at write time and compare them with produced edits rather than relying on read behavior alone.
Takeaways & Limitations
Coverage does not determine the edit when covered sources conflict, and the framework diagnoses fact availability within workloads rather than general planning or reasoning.
Abstract
from arXiv · showhide
Repository-scale coding requires an agent to keep tests, imports, configuration, and migration rules consistent within a bounded context window. We model this as reconstructing a coupled-fact graph: at each edit, a required fact comes from recent context or parametric memory, and the facts covered by neither form coherence debt. We supply and withhold each channel and inject faults across seven models and five harnesses. As expected, no model completes a task on an unseen API with both channels empty, and putting the facts in the prompt restores success. When a rename defeats what models memorized about a real library, all seven fail in the same place, passing and missing the same tests. Availability decides the outcome and distance does not: withholding a fact costs exactly the work it supports, and a supplied fact works as well far from the edit as next to it. Harnesses pay unequal prices for it: configurations that all pass every test differ more than tenfold in tokens consumed because they rebuild the same content at different rates, and spending more recovers nothing when facts are withheld. A missing fact produces wrong work rather than absent work: an agent asked to act acts, fabricating the file or guessing the value, so instruments built on reads look for a hole already filled. How often it says it is blocked instead is a property of the model, from every trial to none. Availability does not settle every edit: where standard and code disagree, agents follow the standard even when it prescribes the worse code, so a stale convention file costs more than no file. Because parametric memory substitutes for reading, on SWE-bench, where models likely know the repositories, reads no longer predict success. Harnesses should keep the facts an edit depends on available when the agent writes, and check that availability against what the agent produces rather than what it reads.
1 Introduction
Repository-scale edits succeed only when the coupled facts they depend on are available at write time, whether from recent context or parametric memory. The paper studies this two-channel account through controlled interventions, showing that missing facts create coherence debt and that read-based measures can miss it.
- Repository-scale edits must remain consistent with coupled facts across files, including tests, imports, validators, configurations, and runtime contracts.Agents reconstruct this structure through reads, edits, tests, and handoffs.
- A required fact is available through recent context or model parametric memory; when neither covers it, the edit proceeds with coherence debt.Recent context is bounded and evicts facts, while parametric memory is fixed for a model and unreliable on novel APIs.
- 299 of 300 matched front-loaded trials reached at least 9 of 12 requirements, whereas no model completed any of 154 closed-book trials.The fictional API migrations began with both channels empty; supplying the exact rules and source files restored performance.
- 66 of 70 renamed-library trials across seven models ended at the same score, each passing the identical 24 of 79 tests.Renaming every API name defeated models’ memorized library knowledge and produced the same failure location.
- Read-derived instruments overstate missing facts by exactly the parametric coverage because model memory can substitute for reading.The paper confirms this consequence by enumeration and argues that harnesses should check fact availability against produced work.
2 Coherence as Edit-Time Coverage
The paper models repository-scale edits as coverage of coupled facts that must remain jointly consistent, with facts supplied by either effective context or parametric memory. Because true fact coverage is hidden in historical logs, the study uses a narrower read-based residency proxy while distinguishing working-set, stale-read, handoff, and speculative-write failures.
- Coupled-fact graph: Repository tasks induce coupled-fact graphs spanning symbols, tests, configuration, imports, migration rules, and invariants beyond static import dependencies.The minimal jointly correct fact set depends on the task oracle, fact schema, and implementation path, so alternative correct patches may require different coverage.
- Edit-time coverage: Success should depend on whether required facts are covered through context or parametric memory, not on which channel supplies them.Uncovered coupled facts create measurable risk, but coverage alone does not guarantee a correct edit.
- Trajectory and thrashing: Thrashing occurs when uncovered edit-time debt survives repeated read–edit–test cycles, so diagnostics use debt per edit and final-quarter mean debt rather than cumulative debt.A long successful trajectory can accumulate more raw debt than a short failure because the working set is bounded and facts enter and leave it.
- Observable proxy: The residency score is a lower-bound proxy that counts unread one-hop import neighbors, not missing facts, and therefore misses dynamic, prose, prompt-supplied, and parametric coverage.The analysis sweeps w ∈ {4, 8, 16, 32, 64, 128}; import-isolated files receive ρw = 1 by convention despite undetected non-import dependencies.
- Prospective measurement: The event stream separates working-set misses, stale reads, handoff gaps, and speculative writes, while measuring parametric shortfall separately because stronger priors need no read event.Rereads repair missing or stale facts, serialization removes cross-worker gaps at a time cost, and event-only estimators cannot observe parametric coverage.
3 Experimental Design
The experiments combine closed-book fictional API migrations, tool-using repository tasks, and controlled secret-fault injections to test coherence under limited fact availability. They evaluate seven model families and multiple harnesses with passed-test or oracle-based outcomes, uncertainty estimates, sandbox controls, and a 100-instance SWE-bench external-validity set.
- Closed-book workloads: Four fictional v1→v2 migrations span Rust, Go, Python, and JavaScript, each with three editable files, a change log, and 12 mechanically checked requirements.The invented libraries and rules make prior exposure highly unlikely while retaining generic programming idioms.
- Closed-book workloads: Seven model families run closed-book trials, while five enter the fully sandboxed front-load matrix; Codex and Gemini are absent from that matrix for startup or quota constraints.The added Pydantic migration plausibly tests parametric knowledge, and its adversarial lexical-shim rename weakens that shortcut.
- Tool-using workloads: 122 matched tool-using trials compare Claude Code, Codex CLI, Aider, and OpenHands on a two-application Pydantic migration with 79 tests and controlled repository bloat.Conditions include lean repositories, skippable documentation, mandatory in-file bloat, and a renamed bloat twin that weakens parametric retrieval.
- Ground truth and fault injection: Synthetic-coherence tasks couple three files through random literals in secret files, allowing withheld motifs to inject faults of controlled size while preserving the remainder of each task.The correct edit is defined by each literal and its upstream dependency, establishing ground truth by construction.
- Outcomes and validity: Outcomes are evaluator scores or passed-test identities for closed-book trials, whereas tool-using success requires the task oracle to pass.Separation uses within-cell ROC AUC with 95% intervals from 1,000 bootstrap resamples; 30 cells were rerun, with 4.8% cross-batch disagreement, and 100 SWE-bench Verified instances were attempted externally.
- Isolation and evidence tiers: The final sandbox denies the complete project root, re-allows only the empty trial directory and required startup configuration paths, and excludes runs with prior leakage.The study distinguishes five large confirmatory contrasts from exploratory supporting comparisons, including a post hoc conflicting-source contrast.
4 Results
Results show that repository-scale coding succeeds when required facts are available, fails exactly where facts are withheld, and is largely unaffected by their distance or the amount spent rebuilding context. Missing facts produce confident wrong edits rather than reliable blocking, while conventions can override correct code and parametric memory can make read-based instrumentation misleading.
- Fact availability: 154 novel closed-book trials scored 0/12 with no workspace or task-specific prior, whereas 213/300 front-loaded trials met every requirement beyond the 9-of-12 threshold.The Wilson 95% upper bound on a nonzero complete-solution probability was 2.4%; the only below-threshold run emitted narrative instead of file blocks.
- Fact availability: Mean line-level Jaccard was 1.00 for Kestrix, 0.99 for Sprocket and Zynet, and 0.79 for Grimwire, where valid Go idioms differ in threading context.Context.This calibration establishes exact floor and ceiling endpoints for later availability sweeps.
- Parametric memory: Renaming a real Pydantic API preserved a shared failure point across all seven families, with trials passing and failing the same tests rather than merely equal numbers of tests.Surviving tests involved locally idiomatic wiring; failures required renamed validator, settings, and configuration rules.
- Context rebuilding: Success stayed flat when a fact was moved as far as 128,000 characters on the tool-using harness or 200,000 on the closed-book harness, while withholding it still floored performance.Across eighteen cells, peak per-turn context varied 1.8× but cumulative input varied 12.8×, from 293,882 to 3,752,134 tokens.
- Coherence debt: Withholding m of eight motifs cost only the work supported by those motifs, with passed tests falling 32.0, 24.0, 16.7, 8.0, and 0.0 across injection levels.Each motif supported four tests, and the median trial lost exactly four tests per withheld motif; maximum deviation was zero tests.
- Failure behavior: Missing facts yielded wrong work rather than absent work: Opus reported the missing file in every trial, Codex CLI and opencode never did, and Haiku fabricated it in three of eight.Agents also guessed values, followed stale written standards over working code, searched elsewhere, or stopped; read-based residency therefore failed to predict success when prompts supplied facts.
5 What the Event Stream Can and Cannot Measure
Event-stream measures cannot reliably infer fact availability: missing facts increase activity, agent-written files can appear covered, and the score may disappear under severe withholding. Directly authored coupling and outcomes provide a more exact alternative, while import-based proxies have limited fidelity.
- Behavioral limitations: Missing facts increase activity rather than reduce it, so read-based residency measures search for a deficit the agent has already filled.Reads of the withheld file rise from 5.5 to 10 to 13 as more facts are removed.
- Behavioral limitations: 1.000 falls to 0.000 when agent-authored paths are excluded, correcting false coverage under deletion.The naive score credits files the agent wrote itself, even when nothing was available.
- Proxy fidelity: 40% recall at 40% precision makes import edges a weak proxy for authored required facts.Directed graphs remove every spurious edge without reducing recall, while identifier resolution reaches perfect recall at 4% precision and serves only as a candidate generator.
- Behavioral limitations: Roughly 60% of full-withholding trials contribute nothing because the agent makes no edit, and the remaining observations are behaviorally selected.Only three of five neighborhood entries per motif are withholdable, setting the total-withholding ceiling at 0.333 rather than zero.
- Alternative measurement: Exact authored coupling and outcomes avoid the graph, window, and event stream required by residency-based measurement.The authors retain residency for historical corpora where better measures cannot be recovered, while stating its limits.
6 Implications and Limitations
The paper argues that harnesses should maintain current, mutually consistent facts coupled to each edit, while emphasizing limits in measurement validity, causal inference, scope, and falsifiability. Its framework primarily targets repository-scale cross-file work rather than single-file or greenfield development.
- Implications for harnesses: Harnesses should keep versioned, mutually consistent facts coupled to the next edit, invalidate them after writes, and explicitly transfer them across workers.The proposed task graph should include tests, instructions, and runtime invariants beyond static imports.
- Latent state and proxy validity: Import-neighbor residency is only a proxy for task coupling, weakening under dynamic dispatch, generated code, or prose rules.Tool logs reveal prompt-supplied facts and compaction only partially, and synthetic fault injection validates the proxy on one constructed family.
- Causal and statistical limits: The conflicting-source result is a perfect separation over 39 trials, but its lower-bound interval remains wide at [0.91, 1.00].The cells also do not separate authority from modality or read order.
- Scope: The framework targets correctness based on cross-file consistency and should explain less for single-file edits, while greenfield development may build its graph during writing.The residency score diagnoses structural fact availability, not planning, reasoning, test quality, or semantic search; six of seven main workloads are migration-shaped.
- Falsifiable boundary: The framework would be weakened by reliable success without required facts, equal prediction from arbitrary and coupled-fact reads, or decomposition effects unrelated to the cut.The first pattern is absent, while existing baselines and intervention cells contradict the latter patterns without fully settling them.
7 Related Work
Prior work studies repository-scale coding through benchmarks, trajectories, context retrieval, long-context memory, decomposition, and context injection. This paper unifies these directions by explaining them through fact availability and coverage at edit time, with familiarity supplying a separate memory channel.
- Repository-scale coding agents: Repository-scale coding research established benchmarks, agent-loop trajectories, change-impact planning, code-graph navigation, and efficient edit and resource representations.SWE-bench scores issue resolution, while related scaffolds expose read–edit–test behavior and exploit repository structure.
- Measuring the context that reaches the patcher: Context-measurement work scores retrieved context, validates restricted-context repair, studies compression, and enforces instruction-file constraints.Trajectory studies diagnose failures after the fact, whereas this paper manipulates fact availability before editing and finds that facts help only while resident.
- Context as managed memory: Long-context research reports positional degradation and shorter effective lengths, motivating memory-hierarchy harnesses and cache-coherence or consistency protocols for synchronization.Other work also investigates internalizing facts into parameters, connecting managed context with memory and synchronization mechanisms.
- Decomposition and context injection: Decomposition and context-file studies partition work by cohesion, bound success through task-graph cuts, and test repository instructions as interventions.The paper claims these approaches are not new, but offers a shared explanation based on coverage of C(i) and task constraints at edit time.
- Decomposition and context injection: The paper links partitioning and front-loading through edit-time coverage of C(i)T via Rt, while repository familiarity supplies the KM channel.This coupling explains when context injection helps and why the same coherence limits affect decomposition.
8 Conclusion … C Event-Only Estimator Enumeration
Repository-scale coding depends on the edit-time availability of coupled facts from context or parametric memory, while disagreement between covered sources and nontransferable diagnostics limits what coverage alone can explain. In the event-only estimator study, read-derived instruments systematically mistake parametrically covered facts for missing ones, doing so on all 1,089 simulator runs.
- 8 Conclusion: Success tracks whether coupled facts are available when the agent writes, not context consumption or the distance of supplied facts from the edit.Coverage alone fails when covered sources disagree, and within-workload failure scores do not transfer to real repositories.
- A Notation and Abbreviations: Coherence debt D(e_i) counts required facts covered by neither effective context R_t nor model memory K_M.The required facts are defined by a coupled-fact graph, while ρ_w measures the fraction of one-hop import neighbors read during the preceding w tool events.
- A Notation and Abbreviations: AUC is reported as a separation statistic in [0, 1], with 0.5 representing chance.The document expands other abbreviations at first use.
- B Workloads and Trial Organization: The supplemental artifact defines closed-book migration success rules and organizes the tool-using corpus into eight matched-intervention blocks.Table A2 records block sizes and harness/model/workload coverage, while Table A3 lists four one-quantity parameter sweeps; later sweeps address withholding, supplied-fact distance, abstention, standard-against-code, and harness spend.
- C Event-Only Estimator Enumeration: The two coverage channels make event-derived coverage estimates blind to facts supplied by parametric memory.Facts supplied through R_t leave reads in the trajectory, whereas facts supplied through K_M leave no event.
- C Event-Only Estimator Enumeration: For required facts partitioned into read-covered R, parametrically covered P, and uncovered U, the event-only estimator reports U ∪ P, overstating misses by exactly |P|.The simulator assigns each fact to exactly one state and writes correct values for held facts and wrong values for uncovered facts.
- C Event-Only Estimator Enumeration: All three estimator predictions hold on all 1,089 runs with no violations.The union-aware estimator exactly recovers uncovered facts; event-only estimates include parametrically covered facts and call every required fact missing when K_M covers a successful edit.
- C Event-Only Estimator Enumeration: The event-only findings characterize an instrument limitation rather than empirical evidence about agent behavior.P-B and P-C quantify what the paper’s read-derived proxy cannot represent, while ρ_w is treated as a lower bound on R_t.
D Five-Event Trace Simulator · E Order Independence and Its Limit · F Symbol-Rename Sweep
The simulator validates the paper’s event model and shows that coherence debt depends on fact coverage at write time, not event order, while stale supplied facts can mislead. A symbol-rename sweep measures resolution errors directly because pass rates collapse distinct failure modes.
- D Five-Event Trace Simulator: 100 matched triplets varied missing, stale, and handoff-stranded facts while preserving identical event-type counts through failure; each failed, reverted, and passed after cause-specific recovery.Each trace used 2–8 required facts and 0–5 irrelevant reads; only source version, provenance, and actor ownership differed.
- D Five-Event Trace Simulator: Retrospective action-observation logs cannot faithfully recover required facts or fact-extraction events, so prospective annotations are necessary for the transition system.The audited fixture had 22 messages, 10 shell commands, three reads, two edits, and three test-like actions, but no explicit coherence events or revert.
- D Five-Event Trace Simulator: The production acceptance trace logged 34 monotone events and was independently replayed to reconstruct four schema versions, the actor ledger, and feedback-linked reversal.The run deliberately injected a missing migration-rule fact and wrong edit; the restored reference edit had debt 0 and passed.
- E Order Independence and Its Limit: Front-loading all context before the first edit sets D(e_i) = 0 everywhere, but it works only when the implementation path is already known.Alternative correct patches require different facts, so order independence holds given an edit sequence but does not determine that sequence.
- E Order Independence and Its Limit: Supplying repository-disagreeing facts is worse than supplying nothing because the written source wins, injecting authoritative error rather than leaving a gap.The account distinguishes front-loadable fact provision from human steering among correct implementations, which changes the applicable context.
- F Symbol-Rename Sweep: 52 trials renamed K of 19 symbol pairs for K = 5, 10, or 19, using either centrality-based or random subsets.Random trials used a fresh seed, causing different trials to contest different symbols.
- F Symbol-Rename Sweep: Pass rates are uninformative under heavy renaming, so the sweep classifies each symbol as correct, stale, mixed, or invented based on the names the agent writes.These four counts remain available even when the artifact fails to run, separating failures that pass rates merge.
G Supplied-Fact Distance … J Direct Withholding Sweep
Across four sweeps, supplied facts remained usable regardless of prompt distance or trajectory length, while withholding facts caused proportional, localized damage. These results bound retention and show that availability—not retrieval position alone—determines performance, although residency measures can misrepresent fabricated recovery.
- G Supplied-Fact Distance: No supplied-fact arm declined with distance up to 200,000 characters, including a harder arithmetic variant without a monotone trend.Qwen’s harder variant scored between 8% and 19%, with its best cell at 32,000 characters rather than zero.
- G Supplied-Fact Distance: Distance within the context window did not degrade an available fact, implicating eviction, re-reading, or ordering rather than position itself.The ceiling results bound large effects only.
- H Invariant-Retention Sweep: No compliance decay appeared across 24, 48, or 96-task trajectories at sixteen invariants, despite context reaching roughly 140,000 tokens.The agent did not re-read the invariant document, so the result bounds this workload rather than proving universal retention.
- I Fault-Injection Sweep: The fault-injection sweep tested additivity: each missing motif should cost the four tests it supports while leaving untouched motifs intact.Each task contained eight independent motifs and 32 oracle-pinned tests.
- I Fault-Injection Sweep: Withholding 0, 2, 4, 6, and 8 motifs produced mean passed-test counts of 32.0, 24.0, 16.7, 8.0, and 0.0.These means closely tracked the linear prediction 32, 24, 16, 8, 0; the widest spread was standard deviation 1.1 at m=4.
- J Direct Withholding Sweep: Direct deletion and equal-length redaction withheld values from k ∈{0, 2, 4, 6, 8} motifs, differing only in whether opening the file was possible.Both manipulations removed the required value while preserving the same task position.
- J Direct Withholding Sweep: Residency was 0.000 at k = 8 after excluding agent-authored paths, preventing fabricated files and read-backs from appearing as recovered context.Without that exclusion, residency returned 1.000 despite no required fact existing.
- J Direct Withholding Sweep: At k = 4, redaction versus deletion yielded AUC 0.59 with a 95% interval of [0.36, 0.81], while the maximum deviation across nine cells was zero tests.At k = 8, AUC was 1.00, but only five and six trials per arm remained scorable and were behavior-selected.
K Harness Spend at Fixed Outcome … O Fidelity of the Import Proxy
Across repository-scale tasks, fact availability—not harness spending or proximity—determines coverage, while missing facts usually prompt action rather than abstention. Agents also follow written standards over conflicting code, and import structure poorly captures the coupled facts edits require.
- K Harness Spend at Fixed Outcome: Withholding k of eight facts makes every configuration lose exactly the withheld proportion, while spend at k=0 still differs by 12.5×.The spread across harnesses never exceeds three points at any withholding level.
- K Harness Spend at Fixed Outcome: 12.8× separates cumulative input across configurations despite every trial passing every test, with 293,882 versus 3,752,134 tokens consumed.Peak per-turn context spans only 1.8×; the difference comes from five versus seventy-nine tool calls and conversation resending.
- L Abstention Across Configurations: Pooled across configurations, 46 trials include 17 blocked, 3 fabricating, and 26 proceeding outcomes when required files are withheld.Blocking requires an explicit inability statement or request; fabrication requires creating the withheld file.
- L Abstention Across Configurations: Haiku is blocked in 13.5% of a separate 96-trial block versus 12.5% here, and placeholder disclosure changes blocking from 8 of 48 to 5 of 48 trials (Fisher exact p = 0.552).The reported difference is not detectable under the stated test.
- M Conflicting-Source Experiment: Ten engineering rules create contestable surfaces where written standards and working code may disagree, with contradictions swept at four and ten while agreeing rules provide control.The surfaces include money rounding, field access, timestamp handling, validation, casing, serialization, logging, naming, and fee-rate sourcing.
- N Standard-Against-Code Sweep: All ten correct-standard trials take every decision, whereas all ten stale-standard trials take none, showing that agents follow the standard even when it prescribes the worse form.The scoring outcome is the share of contested surfaces written the objectively better way.
- O Fidelity of the Import Proxy: Making the dependency graph directed removes every spurious edge without reducing recall, while identifier resolution reaches perfect recall at 4% precision.The workload’s coupled value is a literal that cannot be imported, so it leaves no syntactic trace; such dependencies form the majority of authored edges.
P Semantic Graph Pilot Details
The semantic-graph pilot added fine-grained repository couplings and improved context packing at larger budgets, but it did not improve top-ranked file localization. Conservative resolution and missing dynamic or runtime links limit the graph to a proxy rather than ground truth.
- Graph construction: A Tree-sitter extractor represented files, symbols, tests, imports, decorators, routes, validators, and configuration providers with semantic coupling edges across 21 Python modules.Both representations used the same task text, lexical-anchor scorer, two-hop neighborhood, and cutoffs.
- Localization results: .833→.750: MRR declined, while Hit@1 fell from .667 to .500, showing richer edges did not improve top-ranked file localization.The semantic graph still reached all gold files by rank five.
- Graph construction: The semantic graph contained 445 nodes and 1,521 edges, compared with 21 nodes and 20 edges in the import projection.Its nodes included 21 files, 7 classes, 177 functions, 12 methods, 12 tests, 211 imports, and 5 decorators.
- Packing results: At 12K characters, semantic relevant-file recall was .567 versus .256 for coarse baselines, while relevant-symbol recall was .883 versus .778.Semantic packing used definition-level chunks, whereas baselines packed whole files; the sweep covered 4K, 8K, 12K, and 24K characters.
- Limitations: The resolver conservatively links identifiable repository symbols but omits ambiguous targets and may miss dynamic imports, reflection, injection, monkey-patching, and runtime dispatch.These limitations motivate treating the semantic graph as another proxy rather than ground truth.
Q Measurement and Benchmark Details · R Run-to-Run Stability
The supporting measurements define how residency, engagement, import structure, reading volume, and benchmark execution were evaluated. Across 404 trials, matched full-marks outcomes disagreed between batches on 4.8% of trials, bounding run-to-run noise.
- Q Measurement and Benchmark Details: AUC cells required at least one success and one failure; one-class cells were excluded rather than assigned AUC 0.5.Per-cell intervals used 1,000 stratified bootstrap draws that preserved both outcome classes.
- Q Measurement and Benchmark Details: Residency reached AUC 0.83 on the n=12 lean-versus-code-bloat block, while volume baselines stayed near chance and effort measures separated conditions oppositely.All entries were signed as the AUC at which lean scored higher; the twenty-trial rerun reversed this pattern.
- Q Measurement and Benchmark Details: The proxy import graph was extracted by AST traversal of Python sources, resolving package-relative imports and capturing docstring, conditional, and re-exported imports.Malformed workspaces used a regex fallback, while marked and ripgrep used same-shape language-specific extractors.
- Q Measurement and Benchmark Details: Table A9 reports median bytes of file content read across bloat-placement conditions, separating placement effects from raw reading volume.Pass rates for these same conditions appear in Table 3 of the main paper.
- Q Measurement and Benchmark Details: SWE-bench trials ran under their harnesses, with harness-native transcripts collected when available.Aider runs were scored but excluded from trajectory statistics because this setup exposed no recoverable multi-edit stream.
- R Run-to-Run Stability: 30 benchmark-agent-condition cells were executed twice in separate batches on different dates, covering 404 trials.The batches therefore provided a direct run-to-run stability check.
- R Run-to-Run Stability: 4.8% of matched trials disagreed on full-marks outcomes across batches, bounding noise under the reported contrasts.This disagreement rate is the paper’s stated cross-batch stability measure.
S Execution Environment and Reproducibility · T Ethics and Data Statement
The study documents a reproducible closed-book execution environment, local-model setup, and excluded Codex diagnostic, while releasing code and trial data without human subjects, personal data, confidential code, or repository-derived code.
- S Execution Environment and Reproducibility: Recomputation uses released per-trial ledgers to check condition-discrimination baselines, an independent-fixes control and speedup, estimator enumeration, and a semantic-graph pilot.The recomputation compares derived values against printed values and includes graph statistics and a budget sweep.
- S Execution Environment and Reproducibility: The project repository is available at https://github.com/mpi-dsg/agent-coherence.This provides the stated code location for the study.
- S Execution Environment and Reproducibility: Closed-book trials deny reads beneath the project root while re-admitting each trial’s scratch directory and required command-line interface.The policy is generated per trial and retained with its trial record.
- S Execution Environment and Reproducibility: Isolation is scoped rather than total: paths elsewhere remain readable, while workload-specific answers exist only beneath the denied root.The four novel workloads were hand-authored, and their API rules appear in no other local file or training corpus.
- S Execution Environment and Reproducibility: DeepSeek-Coder-V2-Lite and Qwen3-Coder-30B-A3B-Instruct ran locally through vLLM 0.24 on an H100 PCIe node.DeepSeek has 16B parameters and 30 GB weights; Qwen has 57 GB weights and uses a trimmed 65,536-token context.
- S Execution Environment and Reproducibility: 12/12 was reached in 12 of 24 relaxed-sandbox Codex trials, covering three of six trials per workload.The final policy excluded Codex because its startup read could not complete before the model API call.
- T Ethics and Data Statement: Models were accessed through published inference APIs or self-hosted open-weight checkpoints under their respective licenses, with no human subjects, personal data, or confidential source code.The synthetic workloads were hand-authored for the study and contain no code drawn from any repository.
- T Ethics and Data Statement: The code release includes prompts, raw model responses, evaluation logs, sandbox profiles, and derived per-trial records for tool-using intervention cells.Single-intervention cells are marked exploratory rather than confirmatory, and the release supports recomputation of several reported analyses.