Source-linked AI summary
Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model
Stephanie Jarmak
TL;DR
Reliable coding-agent evaluation must account for execution context and infrastructure, not only model behavior. The paper develops evaluation protocols around variance and operating constraints, finding that infrastructure nondeterminism can shift pass@1 by 2.2 to 6.0 percentage points across 60,000 trajectories.
Problem
Evaluators must determine what execution context and review process can legitimately support because stale context can invalidate reasoning and provenance-focused review diverts attention from behavior.
Method
The evaluation protocol estimates variance from pilot runs and distinguishes accuracy-only capability probes from accuracy-and-cost operating decisions.
Results
2.2 to 6.0 percentage points: single-run pass@1 varied across 60,000 SWE-bench-Verified trajectories, with infrastructure nondeterminism persisting at temperature 0.
Takeaways & Limitations
A single run is one draw, so reliability claims should account for infrastructure nondeterminism rather than treating each score as stable evidence.
Takeaways & Limitations
The evidence is uneven across topics, with some operational questions supported only by cases and recent capability measurements aging quickly.
Abstract
from arXiv · showhide
AI coding agents are commonly evaluated as models but deployed as systems. Their reliability depends not only on model capability, but on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. This monograph examines those boundaries and develops a framework for evaluating and operating coding agents reliably. It synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis. Across this evidence, many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes. Evaluation and operation are treated as a dependency chain in which weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions. The monograph contributes a versioned catalog of 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads; an evidence ledger; a framework for dependency and repair asymmetry across the agent lifecycle; measurements and failure cases from operated agent systems; runnable evaluation and reliability protocols; and five reusable agent skills with evidence maps. Together, these provide a system-level methodology for distinguishing model capability from infrastructure effects, designing defensible evaluations, and building systems that recover safely when components fail. The review is structured rather than exhaustive, evidence strength varies by topic, and results depend on workload and configuration. The methods record which search lanes were executed, which remain unexecuted, and limits on evidence-grading claims.
A.1 Practice catalog index:
The practice catalog index provides a consolidated reference for all 206 records, listing each record’s identifier, practice, chapter, and treatment.
- A.1 Practice catalog index: 206 records are indexed by identifier, practice, chapter, and treatment.The index serves as a consolidated catalog reference for the full record set.
Introduction
The monograph frames coding-agent reliability as a system property governed by dependencies among measurement, execution, retrieval, state, permissions, review, scheduling, and recovery. It offers a structured, explicitly limited evidence base and engineering artifacts for evaluating and operating these systems without treating local cases or downstream confidence as general proof.
- System framing: Weak upstream evidence can invalidate downstream grades, safety decisions, retrieval, review, and configuration choices, creating a dependency chain with repair asymmetry.More samples, judges, or agents cannot repair excluded workloads, inconsistent rubrics, or authoritative empty retrieval results.
- System framing: Coding-agent reliability becomes a system property when runs cross workers or services, compete for resources, mutate shared code, or publish durable effects.The system must preserve intent, authority, state, evidence, ordering, and recovery across independently failing components.
- Scope and limitations: The review is structured rather than exhaustive: ACM, IEEE Xplore, and Scopus searches were not performed, and coverage excludes many indexes, private records, and adjacent model-comparison literature.These absences are disclosed source limitations rather than inferences.
- Scope and limitations: Evidence strength varies by topic, with operational questions often supported only by cases and practitioner reports vulnerable to selection, survivorship, and reporting bias.Author-operated systems provide mechanisms, local measurements, and reproducible failures, but are not independent external evidence or standalone grounds for general recommendations.
- Contributions: 206 reliability records comprise 193 gated practices, including 56 developed in depth, plus 13 research leads, linked through a versioned catalog and machine-readable ledger.The ledger distinguishes direct support, directional findings, corroborating cases, and null or conflicting results.
- Contributions: A minimum reliability pass retains six challengeable artifacts, including paired distributions, cost-quality records, authority boundaries, verified state transitions, failure corpora, and preregistered decision rules.The pass is an entry point rather than a reliability certificate, with a runnable checklist and retained-artifact layout supplied in the repository protocol.
5 Calibrating model graders and separating agreement from correctness … Data and materials availability
Reliable evaluation separates agreement from correctness and treats benchmark scores as outputs of a calibrated, execution-based measurement system. The supplied evidence emphasizes repeated trials, paired comparisons, stronger oracles, workload controls, and external verification before accepting apparent improvements or autonomous correction.
- Run-to-run variance, statistical power, and paired comparisons: +0.0035 was not a supported configuration-level effect after three repeats, with a 95% confidence interval from -0.0005 to +0.0074 and paired t statistic 2.41 below 2.776.The five tasks were paired units, leaving five paired observations and 4 degrees of freedom; the initial apparent advantage came from one unusually low baseline score.
- Run-to-run variance, statistical power, and paired comparisons: 60,000 SWE-bench-Verified trajectories showed single-run pass@1 variation of 2.2 to 6.0 percentage points, even above 1.5 percentage points at temperature 0.Trajectories diverged within the first few percent of generated tokens, so nominally identical evaluations were not deterministic.
- What could the experiment have detected?: Up to 26% of single-score comparisons appeared significant at p < 0.05 despite comparing identical systems, because too few runs estimated method-level variation.Small samples can also exaggerate significant effects, while Colas et al.’s bootstrap false-positive rate was about 10 percent for fewer than ten runs despite a nominal 5 percent level.
- What could the experiment have detected?: 13.2 percent to 7.5 percent was the worked example’s reduction in minimum detectable effect after accounting for the relevant random quantity.The passage warns that this measured reduction is not a general conversion between response count and experimental resolution.
- Compare outcomes item by item: Positive covariance narrows paired-comparison uncertainty without changing the mean difference by removing shared item difficulty.Item-level records also distinguish broad gains, large reversals, and complementary subsets, but pairing is invalid when executable states differ despite a shared task label.
- Benchmark contamination, oracle strength, and workload validity: 36 under-tested tasks and 345 incorrectly passing patches led to re-adjudication corrections of 40.9 percent on the smaller suite and 24.4 percent on the human-screened suite, changing 29 rankings.The same evidence chain includes 13.6 percent PR-issue misalignment across five patterns and eleven scenarios.
- Execution-based evaluation, correction gates, and release tests: 44 percent of tasks were resolved by one model that submitted an answer in every trial, demonstrating that submission is distinct from externally verified correctness.Across 54 cycles, another agent claimed improvement every time while 56 percent of measured deltas were zero or negative, and its self-verdict gate eroded the best state by 19 percent.
- Calibrating model graders and separating agreement from correctness: Agreement is a calibration result, not a correctness verdict.Expert agreement stabilizes reference-label meaning, while stratified sampling, class-specific rates, operating points, review capacity, and failure costs complete the measuring instrument.