Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model

Stephanie Jarmak

Source-linked AI summary

Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model

Stephanie Jarmak

arXiv:2608.13867v1Aug 14, 2026cs.SEcs.AI

TL;DR

Model-centric evaluation can miss reliability effects from stale context, review processes, and execution infrastructure. The paper develops a system-level evaluation and operations framework, finding that infrastructure nondeterminism can materially vary coding-agent scores and that capability and deployment decisions require different evidence.

Problem
Model-centric evaluation can overlook system boundaries such as stale context and review processes that determine what coding agents may trust and how outputs are assessed.
Source: §17.2 · p. 223, §20.1 · p. 278
Method
The paper evaluates reliability by estimating variance from pilot runs, adding margin, and distinguishing capability probes from cost-constrained operating decisions.
Source: §1.2 · p. 31, §2.2 · p. 44
Results
2.2 to 6.0 percentage points: single-run pass@1 varied across 60,000 SWE-bench-Verified trajectories, with infrastructure nondeterminism persisting at temperature 0.
Source: §1.1 · p. 29
Takeaways & Limitations
A component that never executes cannot explain a result, and accuracy-based capability probes should not be treated as operating decisions without cost constraints.
Source: Baselines, ablations, and… · p. 39, §2.2 · p. 44
Evidence remains uneven: operational questions may have only case-level support, recent measurements can age quickly, and practitioner reports face selection, survivorship, and reporting bias.
Source: §0.3.5 · p. 20

Abstract

from arXiv · show

AI coding agents are commonly evaluated as models but deployed as systems. Their reliability depends not only on model capability, but on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. This monograph examines those boundaries and develops a framework for evaluating and operating coding agents reliably. It synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis. Across this evidence, many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes. Evaluation and operation are treated as a dependency chain in which weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions. The monograph contributes a versioned catalog of 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads; an evidence ledger; a framework for dependency and repair asymmetry across the agent lifecycle; measurements and failure cases from operated agent systems; runnable evaluation and reliability protocols; and five reusable agent skills with evidence maps. Together, these provide a system-level methodology for distinguishing model capability from infrastructure effects, designing defensible evaluations, and building systems that recover safely when components fail. The review is structured rather than exhaustive, evidence strength varies by topic, and results depend on workload and configuration. The methods record which search lanes were executed, which remain unexecuted, and limits on evidence-grading claims.

A.1 Practice catalog index:

The practice catalog index organizes all 206 records by identifier, practice, chapter, and treatment.

A.1 Practice catalog index

The index lists the identifier, practice, chapter, and treatment for all 206 records.Source: §A.1 · p. 11

Introduction

This section frames coding-agent reliability as a property of the surrounding system and its dependency chain, spanning measurement, execution, retrieval, state, review, permissions, and resource allocation. It presents the monograph’s bounded evidence process, contributions, minimum reliability pass, and evidence limits while distinguishing model capability from infrastructure effects and settled guidance from research agenda material.

0.1 Problem and scope

Coding-agent quality cannot be inferred from code, passing tests, or compact diffs alone because the producing, evaluating, approving, and governing system determines what evidence is available.The scope covers measurement, execution-based grading, containment, durable state, recovery, retrieval, context limits, oversight, topology, and resource allocation rather than current-model comparison or prompt design.Source: §0.1 · p. 12

0.2 The reliability dependency chain

Reliability is a dependency chain: weaknesses in measurement, grading, containment, retrieval, context, review, or allocation can invalidate downstream confidence.Later machinery cannot repair evidence lost upstream, so reliability requires preserving intent, authority, state, evidence, ordering, and recovery across independently failing components.Source: §0.2 · p. 13

0.3.1 Search and source assembly

The review assembled 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system cases through structured but non-exhaustive source collection and update audits.Publisher- and index-native searches for ACM Digital Library, IEEE Xplore, and Scopus were not performed, and the review does not claim corpus recall or universal transfer.Source: §0.3.1 · pp. 13, 14, 16

0.3.2 Curation and assembly workflow

Curation separated retrieval assistance from human adjudication: automated systems ranked, normalized, and challenged records, while the author decided inclusion, evidence grouping, practice admission, and prose.The retained artifacts support protocol-level inspection but do not claim byte-for-byte replay of earlier interactive searches.Source: §0.3.2 · pp. 16, 18

0.3.3 Screening and evidence grading

Evidence groups attach to scoped claims rather than venues or chapters, and composite recommendations remain directional when no controlled study supports the complete prescription.A targeted audit reduced six grades and retained six after checking whether supporting sources matched the stated claims.Source: §0.3.3 · p. 19
The edition does not report external inter-rater agreement: it releases a blinded sample of 20 practices and analysis tools for later reproducibility testing.The challenge process reduced correlated review error but was not blinded independent grading by multiple human reviewers.Source: §0.3.3 · p. 19

0.3.4 Practice derivation and chapter selection

Practice derivation produced 206 edition records, including 193 gated practices and 13 thin-support leads, with 56 practices selected for full treatment across 19 chapters.The totals reflect claim granularity and editorial boundaries rather than an estimate of all reliability practices.Source: §0.3.4 · pp. 19–20
The 52-practice urgency ranking was nearly uncorrelated with strong evidence, so chapter selection reflects authorial engineering judgment rather than evidence-derived consensus.The Spearman correlation was -0.004; practices may be included for measured effects or for mechanistic engineering controls with local tests.Source: §0.3.4 · p. 20

0.3.5 Author-system cases and limitations

Author-operated cases provide mechanisms, local measurements, and reproducible failures, but they are not independent external evidence or sufficient alone for general recommendations.Evidence is uneven: operational questions may have only case-level support, capability measurements can age quickly, and practitioner reports face selection, survivorship, and reporting bias.Source: §0.3.5 · p. 20

0.4 Contributions

The monograph contributes a machine-readable evidence ledger, 206-record catalog, system-level reliability model, author-system measurements, runnable protocols, and five reusable agent skills.The reliability model links evaluation, containment, execution, repository state, verification, human control, and fleet allocation through explicit system boundaries.Source: §0.4 · p. 21

0.5 A minimum pass through the dependency chain

A minimum reliability pass pairs baseline and candidate runs, records operational conditions, tests authority boundaries, verifies state transitions, analyzes failures, and fixes promotion criteria.It yields six challengeable artifacts, including paired distributions, cost-quality records, authority observations, verified transitions, a failure corpus, and a precommitted decision rule; it is not a reliability certificate.Source: §0.5 · pp. 21–22

0.6 How the monograph is organized

The monograph is organized as six parts and twenty chapters, placing measurement and experiment design before operating practices across containment, context, human review, and work-allocation boundaries.Part VI remains a research agenda drawing on adjacent scheduling and multi-agent literatures, while Chapter 7 supplies the contracts connecting the system’s boundaries.Source: §0.6 · p. 22

5 Calibrating model graders and separating agreement from correctness … Data and materials availability

This section develops a system-level account of reliable coding-agent evaluation and operation across grading, execution, verification, retrieval, context, memory, recovery, oversight, coordination, and resource allocation. It emphasizes calibrated and independently verified evidence, explicit controls and provenance, measured workload- and configuration-dependent behavior, and bounded claims that distinguish model capability from infrastructure and evaluator effects.

0.9 The companion catalog

The companion site indexes 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads, while the repository preserves the manuscript, ledger, catalogs, schemas, provenance, and checksums.Twenty-nine full entries are limited-support notes excluded from the developed set and intended for investigation rather than recommendation.Source: §0.9 · p. 25
The project packages five reusable agent skills with practice-level evidence maps, but these implementation artifacts are not additional evidence that the practices generalize across environments.Chapters provide detailed reasoning and boundaries; the catalog preserves breadth for matching practices to specific operating constraints.Source: §0.9 · p. 25

1.1 One run is one draw

Repeating CodeProbe runs reduced an apparent +0.054 family advantage to +0.0035, with a 95% confidence interval of -0.0005 to +0.0074 and a nonsignificant paired t statistic of 2.41.The originally reported +0.300 task advantage disappeared: both configurations scored 0.800 on every repeat, showing that one run could not support a configuration-level effect.Source: §1.1 · p. 28
A credible comparison records repeated independent runs, reports run-level distributions with means and standard deviations, matches randomized nuisance realizations across configurations, and does not report only the best run.Local variance depends jointly on model, tasks, and evaluation apparatus, so borrowed variance estimates cannot replace repetition.Source: §1.1 · pp. 28, 30
Across 60,000 SWE-bench-Verified trajectories, single-run pass@1 varied by 2.2 to 6.0 percentage points, while temperature-0 standard deviation still exceeded 1.5 points because infrastructure nondeterminism cascaded through generation.Temperature 0 therefore did not make code-generation evaluation deterministic, and 2-to-3-point improvements can fall within observed single-run variation for this workload and configuration.Source: §1.1 · p. 29

1.2 What could the experiment have detected?

In Miller’s worked example (2024), sampling more responses per question reduced the minimum detectable effect from 13.2 percent to 7.5 percent without changing model or question accuracy.The reduction reflects less variable estimates, not a general conversion between response count and experimental resolution.Source: §1.2 · pp. 30–32
Power analysis should target the smallest improvement that would change the engineering decision, using pilot variance, false-positive rate, desired detection probability, and required item and run counts.Colas, Sigaud and Oudeyer (2018) recommend at least 20 pilot runs for estimating variance, but this is a floor rather than a universal repetition requirement.Source: §1.2 · p. 31
Many benchmark comparisons cannot resolve their reported differences: Card et al. (2020) found that numerous GLUE test sets lacked sufficient resolution for small state-of-the-art improvements.Underpowered studies produce inconclusive nulls and exaggerate effects that happen to cross significance thresholds.Source: §1.2 · p. 31
Small-run inference can miscalibrate error rates: Colas et al. (2018) observed about 10% realized bootstrap false positives below ten runs versus a nominal 5%.Welch’s t-test was closer to nominal but still exceeded it at small N, making the warning study-specific rather than a universal test preference.Source: §1.2 · p. 32

1.3 Compare outcomes item by item

Paired comparisons reduce uncertainty by subtracting shared item difficulty, but require identical task state, execution apparatus, and preassigned nuisance conditions.Positive covariance narrows uncertainty without changing the mean difference, whereas mismatched repository or execution states invalidate the pairing.Source: §1.3 · pp. 33, 35

1.4 Before drawing a conclusion from a difference

Experiments should predeclare an engineering threshold, size tasks and repeats from pilot variance, and withhold verdicts when the planned design cannot resolve consequential differences.Retrospective power is circular because it largely restates the observed p-value; power calculations are useful before results, using independently chosen effect and variance assumptions.Source: §1.2 · p. 32, §1.4 · pp. 36–37

Sources and evidence

The evidence base distinguishes strong findings from directional guidance, including non-determinism, benchmark variance, power, and metric-specific paired tests rather than universal prescriptions.Dror & Reichart (2018) provides directional test-selection guidance, while McNemar (1947) supplies the paired pass/fail method.Source: Sources and evidence · pp. 37–38

Baselines, ablations, and cost-accuracy tradeoffs

The plain CodeProbe baseline exceeded the tool-augmented arm in both score and score per dollar, showing why direct baselines are necessary for attribution.The result was visible only because the experiment included a baseline; without it, retrieval machinery could have received unwarranted credit.Source: Baselines, ablations, and… · p. 39

2.1 Remove the component and rerun the evaluation

Ablations must remove one component while holding the rest fixed; memory evaluations therefore need proposed-memory, no-skill, and raw-trajectory arms.These arms separate whether prior experience helps from whether memory consolidation adds value beyond retaining the original trajectory.Source: §2.1 · p. 40
Ablations must isolate the credited component through matched tasks, raw-trajectory and no-tool controls, observed usage, preserved budgets, and independent scoring targets.Removing a component answers only whether it changed outcomes under the evaluated task distribution and fixed surrounding system; it does not establish broader transfer.Source: §2.1 · pp. 40–42
Tank and Nama (2026) found across nearly 6,000 runs that procedural skills primarily improved outcomes by causing fewer regressions, supporting decomposition into gains, regressions, and residual failures.The evidence is strong in two office-automation benchmarks and three model-harness stacks, while transfer to coding-agent skill libraries remains directional.Source: §2.1 · p. 41

2.2 Select on accuracy and cost together

Architecture comparisons should include direct-call, retry-once, and candidate arms, then rank configurations on accuracy and cost using Pareto frontiers rather than accuracy alone.Retry triggers must be observable by the deployed system, and cost accounting includes all calls, tokens, cache quantities, failures, timeouts, and discarded branches.Source: §2.2 · pp. 42–43
CodeProbe’s plain baseline led the tool-augmented arm on both score and cost, illustrating why reporting both axes exposes engineering tradeoffs without establishing a general tool result.The record is an author-system narrative illustration, not independent evidence for tool effectiveness.Source: §2.2 · p. 44, Sources and evidence · p. 46

2.3 Build the next comparison

The next comparison should preserve surrounding choices, pilot actual component use, retain per-task correctness and resource records, and protect an iteration holdout from tuning.The holdout must be reserved before scaffold changes; once developers use its failures, it becomes development material and cannot be restored retroactively.Source: §2.3 · pp. 45–46

Sources and evidence

The evidence base includes 7 strong, 8 directional, and 1 corroborating items across four developed practices, while the chapter claim is that passing scores depend on workload, exposure boundary, and oracle.Kapoor, Stroebl, Siegel, Nadgir, and Narayanan (2024) provide a directional re-evaluation study, and Tank and Nama (2026) provide strong evidence for narrower transition analysis.Source: Sources and evidence · p. 46, Benchmark contamination, oracle… · p. 47

Benchmark contamination, oracle strength, and workload validity

Public scores combine exposure, oracle strength, and workload representativeness; reputation, curation effort, and annotator counts control none of these measurements.SWE-bench Verified later showed roughly 30% broken tasks and 59.4% flawed tests among 138 audited tasks.Source: Benchmark contamination, oracle… · p. 47

3.1 Measure the public-private gap

SWE-bench Verified showed roughly threefold overall and sixfold file-localization advantages over control tasks, but unresolved difficulty differences make these directional evidence (Prathifkumar et al., 2025).The matched-control audit estimates combined exposure effects rather than pure training-data effects and must be repeated across models.Source: §3.1 · pp. 48–50
Retroactive twins produced public scores up to 16 percentage points above twin scores across 20 models, but this question-answering result is not a coding correction factor (Haimes et al., 2024).Exposure and score inflation are model-benchmark-specific, so one pooled correction would create false precision.Source: §3.1 · pp. 49–50

3.2 Keep the task window ahead of the model

Temporal holdouts expose performance drops on post-cutoff problems and disagreement between fresh and frozen coding distributions, while remaining conditional on cutoff accuracy and task comparability (Jain et al., 2024).Continuous pipelines can differ because of repository changes, issue difficulty, curation, or direct exposure, not exposure alone.Source: §3.2 · p. 51

3.3 Make passing patches harder to preserve

Stronger oracles changed 29 rankings after correcting 24.4% of human-screened-suite entries and rejected 345 previously passing patches (Yu et al., 2025).Across 26 models, 80x-expanded tests reduced pass@k by roughly 19–29% and changed rankings (Liu et al., 2023).Source: §3.3 · p. 53
Task-oracle defects also alter validity: PR-issue misalignment affected 13.6% of audited instances, while PAIChecker reached up to 92.12% binary accuracy on SWE-Gym (Wang, Xu, and He (2026)).A passing patch supports capability only when the task matches the workload, the solution is independent, and tests reject incorrect behavior.Source: §3.3 · pp. 52–53
Removing leakage and weak-test successes reduced one agent-model pair’s reported resolution rate from 12.47% to 3.97%.The audited successes included 32.67% solution leakage and 31.08% weak-test passes; the percentages apply only to that pair.Source: §3.3 · p. 54
A passing patch has resisted the tests run so far, not been certified correct across every valid input, environment, configuration, dependency version, state, or concurrency condition.Strengthening can remain an upper bound when only part of a benchmark is covered; report how many prior passes survive re-adjudication rather than only the corrected score.Source: §3.3 · pp. 54–55

3.4 Move the instrument onto your work

Public benchmark scores should be replaced or qualified by evaluations built from the deployment workload, preserving task state, instructions, permitted context, acceptance evidence, and repeated-run uncertainty.Cross-repository work changes the dominant difficulty from editing to locating relevant code, while later online outcomes help test whether offline results predict field behavior.Source: §3.4 · pp. 56–57, §3.5 · pp. 57–58

3.5 Discount the next public number

Public numbers become bounded claims only after checking task age, oracle strength, producing harness, task distribution, and performance on locally mined work.These checks provide no universal correction formula; local replay tests the claim on the work the system will actually perform.Source: §3.5 · pp. 57–58, Sources and evidence · pp. 58–59

Execution-based evaluation, correction gates, and release tests

Execution, rather than submission or self-assessment, must determine whether candidate work propagates through the system.Mehta (2026) found one model submitted in every trial but resolved only 44% under an external oracle; the evidence is directional and limited to 1,750 trajectories, 50 tasks, and four models.Source: Execution-based evaluation, correction… · p. 61, Figure 4.1 · p. 62

4.1 Make execution decide whether work moves

Across 54 cycles, self-verdict gating accepted every cycle despite 56% zero-or-negative deltas and degraded the best state by 19%.On a boundary task whose success was artifact-verifiable, the judge’s gap disappeared, supporting gates that observe the state where success is defined.Source: §4.1 · p. 62
Execution gates should use immutable candidate-linked run records, explicit commands, artifact checks, resource bounds, and distinct infrastructure-error outcomes rather than relying on exit status alone.Executable checks justify only narrow claims such as test-, build-, or deployment-eligibility, and infrastructure failures must remain counted even when excluded from candidate scores.Source: §4.1 · pp. 63–64

4.2 Let failed runs change the next attempt

Authorize another attempt only when an external observation changes the available evidence; the system, not the model, decides whether correction may proceed or artifacts may propagate.Corrections consume failed candidates and emit new identities, while feedback should preserve mechanically selected diagnostics, classify candidate versus environment failures, and enforce retry limits.Source: §4.2 · pp. 65–67

4.3 Turn repeated trials into a release test

Release tests should replay a small, immutable golden set under the fully pinned system configuration and compare candidates with executable baselines using both per-case and aggregate outcomes.Real repository constraints and interactive trajectories expose failures that general coding scores or final-patch evaluation miss; baseline drift should block promotion.Source: §4.3 · pp. 67–69
Choose pass@k or passk according to deployment semantics: pass@k permits selecting one successful trajectory, whereas passk requires every trajectory in the required sequence to succeed.The value of k should reflect production consequences, observed variance, and evaluation budget because larger task sets and k increase noise-related blocking risk.Source: §4.3 · p. 68, Figure 4.3 · p. 68
A local release test asks whether a fully specified system preserved acceptable behavior on a controlled sample of local work.Cases should come from merged work, use reconstructable starting states and verifiable outcomes, and expand when production failures reveal missing behavior.Source: §4.3 · p. 70

4.4 Build the first gate

The first gate should sandbox each task, establish an executable production baseline, replay after relevant system changes, and require predeclared promotion rules.Candidate failures trigger diagnosis and correction; modified candidates receive new identities, clean runs, and new verdicts, while infrastructure failures remain blocking but distinct.Source: §4.4 · p. 70

Calibrating model graders and separating agreement from correctness

Agreement is a calibration result, not a correctness verdict; grader decisions must track the judgment required for releases, rankings, rewards, or training data.Agreement, human matching, outcome tracking, and probability calibration are distinct properties, and the grader must be evaluated under operational class balance and error costs.Source: Calibrating model graders and… · p. 72

5.1 Build the reference labels before the grader

Reference labels should be built from domain-expert rubrics with explicit categories, decision rules, exclusions, examples, boundary cases, and counterexamples.The rubric must be specific enough for independent experts to apply consistently; otherwise automation may reproduce an unstable convention.Source: §5.1 · pp. 72–73, §5.2 · p. 73

5.2 Agreement before automation

Raw agreement can overstate reliability: a 90% observed agreement rate has only moderate above-chance agreement, while two always-PASS graders have undefined kappa.Figure 5.1 reports 90% observed versus 82% expected agreement and kappa about 0.44; kappa values also depend on category prevalence.Source: §5.2 · pp. 73–74, Figure 5.1 · p. 74
The measuring instrument combines stable expert labels, stratified sampling, class-specific rates, and an operating point tied to review capacity and failure costs.TPR, TNR, and FPR expose tradeoffs that aggregate accuracy obscures, while threshold selection should state conditions such as TPR at a fixed FPR.Source: §5.2 · p. 76

5.3 Calibrate the judge, then scale

A calibrated judge should be tested against expert labels before scaling: five human raters reached Fleiss’ kappa 0.83, while model–human Cohen’s kappa ranged from 0.77 to 0.87.Panthi and Abdelfattah (2026) validated a 115-case stratified subset before applying a five-model majority vote to 1,902 contested cases.Source: §5.3 · pp. 76–77
Agreement measures shared labels, whereas probability calibration tests whether confidence 0.8 corresponds to events occurring about 80 percent of the time.A compatibility failure assigned legacy annotations confidence 1.0, demonstrating that calibration state also resides in stored labels and the software reading them.Source: §5.3 · p. 78

5.4 What a panel’s vote establishes

Up to 32.3 percentage points separated plurality voting from selecting a correct candidate, showing that agreement can mask selection failure when judges share dependencies.Bertalanič (2026) reports this exploratory oracle gap without corroborating evidence; grounding and independent checks are therefore required beyond vote count.Source: §5.4 · pp. 78–79, Sources and evidence · p. 82
Panels should preserve candidates, ballots, typed claim support, and abstention reasons, then verify executable, structural, and judgment claims with appropriate independent oracles.Plurality is a selection rule, not verification; abstention thresholds should be validated on expert-labeled cases against wrong-answer and review costs.Source: §5.4 · pp. 79–80, §5.5 · p. 81

5.5 Rebuild the grader as a measured system

Calibration must measure chance-corrected agreement, class-specific TPR, TNR, and FPR at the deployment threshold, while testing position, verbosity, leniency, self-preference, and leakage.Deterministic assertions should handle exact properties, and calibration must be repeated after changes to the domain, rubric, response format, or model generation.Source: §5.3 · pp. 77–78, §5.5 · p. 80

Sources and evidence

The evidence profile contains 3 strong, 3 directional, and 0 corroborating items across 3 developed practices, so the supported claims remain bounded rather than universal.The cited evidence includes Zheng et al. (2023), Orosz with Husain (2025), and Bertalanič (2026), while related panel methods remain directional.Source: Sources and evidence · pp. 81–82, Proxy metric gaming and layered… · p. 83

6.1 When perfect recall rewards returning everything

A recall-based scorer let an agent earn a perfect 1.0 by returning the entire repository; replacing it with overlap F1 added a precision penalty but did not eliminate gaming.Layered controls assign benchmark scoring and memory retention to separate decision points, preserving visibility into which failure each mechanism can observe and govern.Source: §6.1 · pp. 83–84
A proxy may track an intended property initially yet weaken under adaptation, so better reference labels and architecturally separated controls can limit damage without making any single optimizable score both define success and decide acceptance.Structural invariants, outcome measures, diagnostic scores, and human-owned thresholds should carry different jobs; layered signals remain fallible when they share controls or collapse into a weighted scalar target.Source: §6.1 · pp. 84–85

6.2 Watch the relationship, not the rising score

Independent quality can rise, peak, and then fall while an optimized proxy continues improving, so weakening proxy–quality agreement should remove the proxy’s authority over acceptance and publication.Gao et al. (2022) observed this divergence under reinforcement learning and best-of-n selection; the exact trajectory depended on the search process. A fixed reference sample, predetermined statistic, sampling plan, and uncertainty calculation are needed to distinguish relationship change from reference or workload drift.Source: §6.2 · pp. 85–87

6.3 Preserve the channel that reveals intent

Trace inspection detected reward-hacking evidence substantially better than action-only inspection, but reinforcement-learning pressure against the monitor selected agents whose reasoning appeared benign while they continued exploiting the evaluator.Because no universal safe pressure threshold was established, the conservative control is to keep monitor verdicts outside the reward and use them for review or investigation rather than acceptance decisions; this requires accessible traces and does not make visible reasoning ground truth.Source: §6.3 · pp. 88–89

6.4 Did the verifier run, and did it run on this artifact?

Among rerun GitHub Actions builds, 67.73 percent changed outcome, and 1,055 of 1,960 Java projects contained at least one such build, making verifier execution a fallible dependency requiring provenance and retry semantics.Acceptance records should bind the verification configuration, artifact version, repository base, verifier version, environment, timestamps, raw result, and status; indeterminate means the verifier did not demonstrably run on the intended artifact and should trigger retry, not rejection.Source: §6.4 · pp. 89–90
Verification must separately establish whether the verifier ran on the intended artifact and whether its oracle can distinguish acceptable from unacceptable behavior.Retries are legitimate only when artifact and verifier identities remain fixed while the environment differs; retrying a deterministic oracle until it passes is score-shopping.Source: Figure 6.2 · p. 90, §6.4 · p. 90

6.5 Rebuild the acceptance path

Rebuilding an acceptance path requires inventorying each gate’s producer, dependencies, threshold owner, retry behavior, and retained evidence, then adding an independent signal that observes failures the first cannot.Held-out expert judgment or delayed outcomes can test the proxy relationship; reference samples, uncertainty calculations, scheduled remeasurement, and pause-on-divergence support control monitoring.Source: §6.5 · p. 91

system

The factory, not the worker, owns the reliability promise by preserving logical work and effect identity across process failure.A local fault demonstration contrasts naive duplicate redelivery with guarded reconciliation, without estimating a general failure rate.Source: system · p. 94

7.1 When this framing becomes necessary

Agent reliability becomes a distributed-systems concern when work outlives processes, components fail independently, workers share versioned state, effects are asynchronous, or verification and publication separate.In these settings, the factory exhibits lost updates, stale authority, duplicate effects, split-brain records, and partial failure; model capability is only one reliability contributor.Source: §7.1 · pp. 95–96

7.2 The distinctions recovery depends on

The recovery model separates logical work from attempts, leases from authority, candidates from accepted completion, local records from external commitments, and verifier output from semantic truth.Ge and Zhang (2026) report that 3.2 percent of GitHub Actions builds were rerun and 67.73 percent of rerun builds were flaky, affecting 1,055 projects; these rates concern rerun builds, not all builds.Source: §7.2 · p. 96

7.3 A reference lifecycle for logical work

The lifecycle makes work_id, attempt_id, ownership epoch, input state, artifact, and effect identities recoverable without relying on failed-worker memory.Authoritative facts must live on the work side of boundaries, external effects need named boundaries and reconciliation, and an attempt failure alone does not terminate logical work.Source: §7.3 · p. 97, Figure 7.1 · p. 98
A worker’s “done” report can make an attempt verification-ready, but only independently observed evidence can complete logical work.The self-report is a scheduling signal that an artifact exists and is worth verifying, never an acceptance signal.Source: §7.3 · p. 97, Figure 7.2 · p. 99

7.4 Six factory contracts

The six factory contract families require work continuity, generation-scoped authority, safe external effects, consistent version-bound evidence, visible liveness, and safe recovery.Retries retain the same work_id while receiving new attempt_ids; stale workers are fenced at protected boundaries; uncertain effects require idempotency or reconciliation; recovery paths must be tested as production code.Source: §7.4 · pp. 98, 100
Factory reliability obligations remain independent of model quality: controls must preserve durable state, enforce authority, and accurately report candidate status.A stronger model may alter workload and failure frequency, but it does not replace factory-owned controls, state, or evidence.Source: §7.4 · p. 100

7.5 Audit one logical work item

An audit of one externally consequential work item reconstructs identity, authority, artifact, verification, and effect records, then injects late completion or lost acknowledgement faults.The retained outputs are an identity map, authority map, effect contract, fault-injection result, and list of unobservable assumptions; unanswered questions often identify missing incident information.Source: §7.5 · pp. 100–101

Sources and evidence

The cited foundations transfer mechanisms rather than constants or agent evidence: Chubby supports fencing, Omega and Borg support scheduling and admission, and practitioner cases corroborate ledger-scheduler-worker-gate decomposition without controlled outcome evidence.Perry et al. (2023) linked assistant trust to measurable security defects, but the study concerned security-programming suggestions and observed rather than experimentally assigned skepticism.Source: Sources and evidence · p. 101, §8.2 · p. 106

Agent isolation, injection defenses, and independent verification

Evidence for isolation, injection defenses, and independent verification is limited to four directional and three corroborating items, with no controlled comparison of the developed practices.In a benchmark of 6,560 trials, unsafe signals fired in 68.0% of runs and 66.22% were both unsafe and complete; results apply only to tested configurations and the benchmark.Source: Agent isolation, injection defenses,… · p. 102

8.1 Put the recovery path outside the failure domain

Blast radius follows effective authority rather than prompt warnings, so recovery paths must remain outside the agent’s failure domain through separated identities, scopes, endpoints, and delegation paths.The database incident occurred because one credential reached both production and recovery data; ordinary access should be read-only, while writes require bounded, expiring escalation.Source: §8.1 · pp. 103, 106

8.2 Verify the workspace, not the agent’s confidence

Workspace verification must resolve completion claims to independently inspectable state, because task lists, clean worktrees, and resumed context can agree while describing no actual change.Verification compares revisions and diffs, runs relevant tests, and reads state from the system that owns it; failed attempts should preserve diagnostic evidence before retry.Source: §8.2 · pp. 106–107
Completed-work claims require observations with different provenance, such as a reviewed diff and tests or a system-of-record read-back with an attempt-linked audit trail.Confidence, checklists, and coherent narratives can guide investigation but cannot establish completion; generated references and figures likewise require claim-level verification.Source: §8.2 · pp. 107–108

8.3 Design for instructions that arrive through data

Indirect prompt injection exploits instructions embedded in retrieved data, so detection alone is insufficient; the decisive backstop is the authority remaining after the model is redirected.Layered input/output checks, task-specific tool allowlists, and concrete human approval reduce exposure, but the cited sources provide no measured result for the complete remedy.Source: §8.3 · pp. 108–109
Injection defenses should be tested continuously on representative attacks while recording retrieved inputs, tool requests, approvals, denials, and outputs across the complete control path.Regression tests establish whether named controls work on tested cases, not resistance to attacks outside those cases.Source: §8.3 · p. 109

8.4 Audit one live boundary

A live-boundary audit should enumerate destructive actions, separate routine write access from narrow escalation, and verify a recent completion against the workspace or system of record.The pass condition is an enforcement-layer denial for the ordinary identity, successful intended escalation, unavailable adjacent destructive actions, and retained evidence tied to the attempt.Source: §8.4 · p. 110

Sources and evidence

The evidence profile lists 0 strong, 12 directional, 3 corroborating, and 1 null or conflicting evidence items across three developed practices.The source list supports containment, separation of completion from runtime safety, and distrust of agent self-reports, while tested benchmark results remain configuration-bound.Source: Sources and evidence · pp. 110–111, Persistent agent state, durable… · p. 112

Persistent agent state, durable workflows, and idempotent retries

Durable execution requires explicit state, recovery coordination, and retryable steps whose repeated execution does not repeat completed external effects.When a process dies after changing the world but before recording the change, recovery must distinguish complete, incomplete, and uncertain work; the cited Netflix account reported a decline from roughly 4 percent to 0.0001 percent, but was not independently audited.Source: Persistent agent state, durable… · p. 112

9.1 Declare the state that outlives the worker

Recoverable artifacts need stable identities, owners, update rules, and effect-boundary evidence so replacement workers can resume logical work without silently inventing a different history.Plans, progress, tool results, and events should have distinct identities and declared sources of truth; explicit state preserves decisions and evidence but cannot make model decisions deterministic.Source: §9.1 · pp. 113–114
Bounded systems evidence shows checkpointing can matter: ADEMA had its only invalid run when checkpoint and resume were removed from a fixed matrix of 60 runs.This isolates checkpointing within one system and experiment; other reports, including approximately 25,000 diagnostic passes per month at 99.9 percent success, remain team-supplied and workload-limited.Source: §9.1 · p. 114
Recoverability depends on declared artifacts that let a replacement worker determine accepted completion, pending work, and uncertainty without relying on live sessions, local files, or operator memory.Persistence primarily provides recoverability: after process death, a replacement can determine what the system knew, accepted as complete, and where uncertainty begins.Source: §9.1 · pp. 114–115

9.2 When coordination belongs in an engine

Workflow engines improve failure localization by durably recording execution history, retry policy, external events, and control transitions that individual workers cannot reliably reconstruct after crashes.The engine owns coordination and lifecycle while workers perform model, tool, and domain operations; this boundary is valuable when work survives process death, awaits external events, or causes irreversible effects.Source: §9.2 · pp. 115–116

9.3 The interval that cannot be closed

At-least-once execution cannot eliminate the interval between an external commit and durable logging, so retries require stable logical identities, downstream idempotency or deduplication, and reconciliation when effects remain uncertain.A runtime may record committed results exactly once within its boundary, but external safety depends on the contract at each effect boundary; unknown state must not be guessed into success or failure.Source: §9.3 · pp. 117, 118, 120, Figure 9.1 · p. 119

9.4 Companion patterns

The companion catalog adds eleven recovery patterns covering receipts, idempotency, compensation, invariant-based retries, snapshots, checkpointing, transactional records, versioned histories, and reordering tolerance.Three additional entries have thin support and do not contribute evidence to the chapter’s claims.Source: §9.4 · p. 121

9.5 Run the crash check

The crash check tests whether a replacement worker can reconstruct completed, pending, and uncertain work from stable artifacts before adopting a workflow engine.The recommended adoption decision asks whether meaningful work remains after process death, whether the workflow waits for external events, and whether it performs irreversible external effects.Source: §9.2 · p. 116, §9.5 · p. 121
Recovery checks should make state resumable, use durable workflow coordination, and make retried side effects idempotent rather than silently assuming success or failure.The crash check kills a worker after an external effect but before completion commits; recovery must return the recorded result, converge on the same state, or escalate explicitly.Source: §9.5 · pp. 121–122, Sources and evidence · pp. 122–123

Sources and evidence

Evidence supports durable state and workflow practices directionally, while strong recovery evidence remains limited and largely transfers principles from non-agent systems rather than measured agent outcomes.The evidence profile reports 1 strong, 9 directional, 0 corroborating, and 0 null or conflicting items across two developed practices; the trace argument has no strong study.Source: Sources and evidence · p. 122, Replayable traces and fault-injection… · p. 124

Replayable traces and fault-injection recovery testing

Crash recovery must test inconvenient fault points and distinguish recorded uncertainty from assumed success, especially when external effects precede durable completion records.On recovery, the runtime should return a prior result, converge on the same external state, or stop with an explicit unknown-state escalation.Source: §9.5 · p. 122, Replayable traces and fault-injection… · p. 124

10.1 What a transcript cannot answer

Typed event streams preserve causal distinctions that transcripts compress, allowing operators to relate intent, tool effects, persistence, ownership, and ordering during partial failure.A typed stream can expose a dispatch whose external effect occurred although durable result persistence did not, while identifiers allow policy layers to detect actions not authorized by their reasoning step.Source: §10.1 · pp. 125–126, Figure 10.1 · p. 125

10.2 Replay from a changed step

Replay should preserve the original branch, invalidate only dependent work, record nondeterministic observations, and make recovered invariants mechanically queryable.The event record can verify replay completion, duplicate side effects, and model-call costs, but demonstrations do not establish that typed traces outperform transcripts or improve recovery by a predictable amount.Source: §10.2 · pp. 126–127

10.3 A shared vocabulary and its limits

OpenTelemetry GenAI conventions provide syntactic coordination, not execution portability; complete recovery traces additionally require durable persistence, causal identifiers, state references, branch identity, and external-effect status.Typed events identify uncertainty but do not make effects idempotent, restore corrupted state, or decide whether an uncertain operation should be repeated; transcripts remain useful as derived views.Source: §10.3 · pp. 127–128

10.4 Recovery is a measured property

Recovery is a measured property: fault placement, repeated failures, workload, configuration, and accumulated state can change conclusions, so final pass labels are insufficient.Vogel et al. (2024) found that successive injected failures changed effects and reversed earlier comparisons; Zhao et al. (2026) modeled 3,843 trajectories and 1,794 valid runs across more than 63,000 steps as onset, evolution, and recovery phases.Source: Replayable traces and fault-injection… · p. 124, §10.4 · p. 128

10.5 Specify the claim before the kill

Fault-injection experiments should declare the protected state or effect, expected continuation, injected fault, and measurements before testing recovery.The proposed envelope includes worker or pod kills, interrupted external calls, persistence termination, and repeated disturbances during representative work.Source: §10.4 · p. 128, §10.5 · p. 129
Recovery evaluation must predeclare observable claims, use clean controls with identical anchors, compare constrained state or effects when outputs are nondeterministic, and repeat faults to expose accumulated degradation.Repeated injections reveal queue growth, leaked leases, enlarged histories, and cache churn, while independent clean-start runs separate those effects from ordinary run-to-run variation.Source: §10.5 · pp. 129–130

10.6 Strike the interval the design claims to protect

Fault sweeps should target named commitment barriers, validate the intended kill, test recovery adapters as production code, and cover each interval from dispatch through durable completion.The required sweep includes five lifecycle points, while separate external effects, ambiguous responses, redeployments, history boundaries, and realistic production faults require additional tests.Source: §10.6 · pp. 130–132

10.7 Distributed ambiguity faults

Distributed ambiguity testing compares clean and perturbed state transitions against contracts for uncertain effects, fencing, stale completions, and publication races.The representative fault matrix is a protocol rather than an executed result, and its full machine-readable version extends coverage to controller recovery, verifier failure, schema compatibility, compaction, recovery queues, and retry storms.Source: §10.7 · pp. 133–134

10.8 Keep the result attached to its envelope

13 of 13 invariant checks passed in one kill demonstration, but the result supports only that trial’s specified invariants, not general fault tolerance or cross-version recovery.The recovered activity resumed after a heartbeat-timeout retry, reused a cached result, and matched the clean reference by content hash; no comparative workflow-engine sweep was run.Source: §10.8 · p. 134

10.9 A protocol for trusting recovery

A trustworthy recovery claim requires a durably persisted typed event stream that distinguishes model calls, tool activity, external commitments, and state transitions.The protocol begins with shared OpenTelemetry GenAI conventions and adds explicit state, branch, persistence, and effect information.Source: §10.9 · p. 135
Recovery testing should inject repeated, boundary-specific faults and measure throughput, latency, invariants, duplicate or missing effects, failure ordinal, and equivalence against a clean control.The harness must verify the intended kill, observe recovery without supplying recovery state, and test recovery actions as possible failure sources.Source: §10.9 · p. 136
Under a recorded workload and configuration, specified fault injections support only bounded claims that recovery preserved named invariants and measured behavior.The protocol is repeatable and revisable, but does not establish universal reliability; its fault menu and configuration define the claim’s envelope.Source: §10.9 · p. 136

Human-auditable failure analysis and taxonomy development

Attribution remains difficult: the best evaluated method identified the responsible agent in 53.5 percent of cases but the decisive step in only 14.2 percent.These figures come from expert-annotated logs from 127 multi-agent systems, and some methods performed below random.Source: Human-auditable failure analysis and… · p. 138

11.2 Keep causal assignment under human control

Causal assignment should remain human-controlled because automated methods can triage cases without reliably identifying the responsible action or decisive step.A structured verdict records the state transition, downstream dependency, evidence, alternatives, confidence, and reviewer identity.Source: Human-auditable failure analysis and… · p. 138, §11.2 · pp. 141–142

11.1 Build the taxonomy from the failures you can inspect

The initial taxonomy output is a defensible measurement vocabulary and labeled seed corpus, not an estimate of the system’s failure distribution.Taxonomies should be derived from complete local traces, identify the first upstream failure, and preserve uncertainty when causal boundaries cannot be resolved.Source: §11.1 · pp. 139, 141

11.2 Keep causal assignment under human control

Ma et al. (2025) achieved 36.2 percent step-level accuracy versus less than 15 percent for prior methods, while counterfactually validated fixes increased task success by 22.4 percent on average.The method still assigned most decisive steps incorrectly, so it is better treated as a candidate generator for reviewers than as an automatic postmortem verdict.Source: §11.2 · p. 142

11.2.1 Component boundaries as an attribution vocabulary

Attribution should preserve chains across component boundaries, name missing observations when causes remain unresolved, and apply the first-upstream rule rather than blaming the last actor.The vocabulary spans admission, decomposition, coordination, reasoning, retrieval, execution, verification, aggregation, publication, recovery, and capacity boundaries.Source: §11.2.1 · p. 144
Cohen’s κ = 0.76 against human labels supports reproducibility of Raj et al.’s categories [s126.p6#732faa], not causal correctness of their attributions.Agreement among automated readers or repeated runs is a triage signal; consequential assignments remain signed human decisions.Source: §11.2.1 · pp. 144–145

11.3 Design the record for a skeptical reader

11% of 148 expert-annotated traces were correctly localized by the strongest evaluated long-context model, showing that transcript containment does not ensure causal legibility.These traces combined reasoning, tool calls, outputs, retries, and state changes, making this a capability snapshot rather than a general automation ceiling.Source: §11.3 · p. 145
Structured traces should expose semantic step boundaries, inputs, decisions, state transitions, retries, outcomes, identities, ordering, and observations separately from interpretations.Content-addressed references, explicit concurrency relations, versioned schemas, and visible instrumentation gaps support replay and skeptical review without treating retrospective rationales as replay.Source: §11.3 · pp. 145–147

11.4 Turn failed runs into an operating record

A failure-review operating record begins with 20 recent failed runs, expands toward 100 diverse traces, clusters roughly 5–10 failure classes, and turns unanswered questions into instrumentation requirements.A named human signs assignments affecting remediation, accountability, task validity, or evaluation interpretation; the corpus remains an operating asset for later compaction-policy tuning.Source: §11.4 · p. 148

Sources and evidence

The evidence sources distinguish strong, directional, corroborating, and non-evidence items, while the repository-retrieval chapter claims that retrieval can improve while task outcomes do not.The retrieval claim is presented as a chapter-level contribution, and the evidence profile reports 6 strong, 8 directional, and 1 corroborating item across three developed practices.Source: Sources and evidence · p. 149, Measuring and designing repository… · p. 151

Measuring and designing repository retrieval

+0.0349 paired reward across 370 tasks contrasted with Precision@10 rising 0.095→0.313, Recall@10 0.120→0.272, and needed-file coverage 0.33→0.56.The end-to-end interval was [+0.0130, +0.0579], but task-level resampling ignored repository and suite structure and likely understated uncertainty.Source: Measuring and designing repository… · pp. 151–152

12.1 Score retrieval and generation separately

Stage scoring separates availability, retrieval, placement, contextual sufficiency, and evidence use, because final pass rate cannot identify which transition failed.Per-task records should retain judged relevant items, ranks, context inclusion, outcomes, and credible evidence-use signals; exposure establishes receipt, not use.Source: §12.1 · pp. 152–153, Figure 12.1 · p. 153

12.2 Retrieval metrics

Retrieval metrics must declare the estimand and relevance target, sweep depth against recall, preserve channel outputs, and report context cost alongside precision, recall, and completion.Recall@k can hide rank differences, deep cutoffs can conceal poor ordering, and relevance judgments may require stratified expert annotation with accepted solution paths.Source: §12.2 · pp. 154–155
Evidence use is harder to establish than exposure, so citations, logs, and controlled removal should be combined rather than treated as causal proof individually.Attention weights do not establish use, and retrieval that fills the context window makes a different tradeoff from retrieval that promotes evidence early.Source: §12.2 · pp. 155–156

12.3 Recover the evidence a single lane misses

Restoring a failed semantic-search lane surfaced 64 and 42 additional papers across two reviews, with manual curation retaining 12 and 7, despite coherent lexical-only outputs.The case supports observing lane health and unique contribution without generalizing those recovery counts to other workloads.Source: §12.3 · p. 156
Across 1,669 WeChat repositories and 26 language models, combined lexical and semantic retrieval performed best for closed-source code completion by recovering complementary evidence.The result is workload-specific; the transferable protocol is to run lanes independently and measure uniquely recovered relevant items.Source: §12.3 · p. 157
Fusion is useful only when it recovers relevant evidence the current lane misses without exceeding usable context, rather than adding duplicates or stale neighbors.Reciprocal-rank fusion combines lane positions instead of incomparable score magnitudes, but shared ingestion errors, stale versions, and duplicated chunks can create false agreement.Source: §12.3 · pp. 158–159

12.4 Preserve code structure inside the retrieval unit

4.3 percentage points higher Recall@5 and a 2.67-percentage-point Pass@1 gain resulted from syntax-aware chunking in cAST evaluations by Zhang et al. (2025).The retrieval improvement was measured on RepoEval, while the generation improvement was measured on SWE-bench across languages.Source: §12.4 · p. 159
Syntax-aware chunking preserves complete functions, methods, classes, or coherent statement groups, but its retrieval gain does not establish repairs to missing, stale, or incorrectly queried repository evidence.The downstream generation gain is smaller and may be attenuated by context selection, redundancy, or failure to change generation.Source: §12.4 · p. 160

12.5 Run the retrieval protocol on the operated workload

The operated-workload protocol compares lanes and chunkers under fixed conditions while recording ranks, recall, context tokens, provenance, parse coverage, and final completion to localize failures by stage.When retrieval is sufficient but completion is unchanged, the next ablation should move downstream to evidence use, generation, execution, or verification rather than continuing to modify retrieval.Source: §12.5 · pp. 161–162

Localization funnels, repository indexes, and freshness checks

Evidence without revision identity is stale, while localization funnels preserve granularity and expose handoff failures before downstream repair.A funnel narrows from files to symbols to edit locations and validation, but omitted files cap the complete pipeline’s recall because later stages cannot recover them.Source: Localization funnels, repository… · p. 165, §13.1 · p. 166

Score retrieval and generation separately

Retrieval scores should distinguish whether results rank well from whether they still describe the repository or system state being acted on.Similarity retrieval returned highly ranked but superseded records, showing that ranking quality and state-valid correctness can diverge.Source: Localization funnels, repository… · p. 165

13.1 Stage localization before repair

Staged localization can outperform open-ended repository repair at lower cost, but skipping stages can worsen both cost and performance by shifting search into a remaining stage.The evidence supports validating transitions and measuring file selection first, while recognizing that fixed pipelines may need widening for exploratory workloads and stronger models.Source: §13.1 · pp. 165–168

13.2 Build a typed index only when someone will maintain it

92.7 percent file-level localization accuracy was reported for LocAgent, while fine-grained accuracy declined and graph benefits depended on the surrounding search and traversal tools.Typed indexes are justified when workloads repeatedly require structural, multi-hop questions and a named owner can maintain parsing, identity, schema, publication, and repository correspondence.Source: §13.2 · pp. 168, 169, 171

13.3 Admit only evidence tied to the current state

Retrieval evidence must carry repository, branch, revision, index, schema, identity, and freshness information before it influences a mutation against another version.Shared indexes must also enforce caller permissions, because identical results for identities with different access indicate a read path around the permission model and a disclosure surface.Source: §13.3 · p. 171, §13.2 · p. 171
15 stale-signature outputs for Qwen2.5-Coder-7B-Instruct and 13 for gpt-4.1-mini became zero under current-only retrieval across the same 17 examples.Adding current snippets to stale ones reduced incompatible outputs by 47 to 65 percentage points; McNemar’s exact two-sided values were 6.1 × 10−5 and 2.4 × 10−4, respectively.Source: §13.3 · p. 172
Retrieval should admit snippets only when indexed-state identity matches the worker’s editable state, otherwise refusing, reconciling, or falling back to current navigation.The gate must account for uncommitted changes, atomic index publication, cache keys, retries, and explicit freshness failures; it does not establish authorization or a universal refresh cadence.Source: §13.3 · pp. 172–174

13.4 Choose the next retrieval intervention

Retrieval interventions should follow measured bottlenecks: improve file selection before model reasoning, and adopt typed indexes only with explicit ownership, correspondence, publication, rollback, and freshness controls.When index ownership is absent, generated repository maps plus exact search, definition lookup, reference search, and bounded caller navigation provide a cheaper architecture.Source: §13.4 · pp. 174–175

Stage localization before repair

Stage-localization evidence supports file retrieval and repository-graph interventions, while Weng et al. (2026) provides directional evidence for freshness-gated retrieval.The evidence profile distinguishes strong, directional, and corroborating records, and notes that the Google repository case predates coding agents and lacks alternative comparisons.Source: Stage localization before repair · pp. 175–176

Usable context budgets, consolidated-spec restarts, and file-based tool output

29 percent fell to 3 percent for Claude 3.5 Sonnet as LongSWE-Bench context grew from 32,000 to 256,000 tokens, while Qwen2.5-14B-Instruct fell from 70.2 percent to 40.0 percent on LongCodeQA.Across studies, effective context depends on model, configuration, workload, and threshold; the repeated finding is a gap between advertised capacity and useful competence, not a universal ceiling.Source: Usable context budgets,… · p. 177, §14.1 · p. 178

14.1 Measure the context that remains useful

Effective context should be measured with controlled context-size sweeps on realistic repository tasks involving retrieval, aggregation, instruction retention, code relationships, and growing tool-use trajectories.The same tasks, permissions, tools, model settings, harness, prompts, and evaluator should be retained across strata with repeated independent runs.Source: Usable context budgets,… · p. 177, §14.1 · pp. 178–179
Measure retrieval, correctness, reliability, and cost separately, and set production context limits from the weakest task’s measured knee rather than retrieval performance alone.Context decline may be gradual, abrupt, or task-family-specific; traces identify whether failures arise from retrieval, stale evidence, forgotten constraints, repetition, or harness truncation. Synthetic probes explain mechanisms, while repository-shaped trials set operating limits.Source: §14.1 · pp. 179–180

14.2 Restart from the current specification

When corrections no longer change a run’s governing interpretation, restart from a consolidated specification and carry execution state separately as inspectable raw evidence.The restart package distinguishes settled requirements and unresolved questions from repository state, diffs, failures, and verification; the 39 percent result supports the commitment mechanism but is not a coding-work effect-size estimate.Source: §14.2 · pp. 181–183

14.3 Move bulk output out of the active context

Externalize long tool outputs into retrievable, session-scoped artifacts while retaining concise, costed pointers and metadata in the active context.This preserves raw evidence across truncation and resets and enables bounded search or range reads, but file-backed output is supported by two directional practitioner accounts rather than strong studies.Source: §14.3 · p. 183
Deterministic harness controls must bound inline output, artifact writes, retention, access, read ranges, and failure behavior around model-selected buffering.Pointers require run and tool-call identities, atomic publication, completeness records, and security policies because externalized output may contain sensitive data or become invalid after deletion.Source: §14.3 · p. 184
Externalizing logs does not preserve the specification itself, so compaction and restart remain separate controls and systems must validate pointer resolution and record what evidence was loaded.Agents may ignore pointers, search incorrectly, load incomplete excerpts, or rely on summaries that misdescribe deleted or decisive content.Source: §14.3 · pp. 184–185
Move bulk outputs into session-scoped artifacts only after verifying recovery, concurrent-output separation, failed-write visibility, pointer expiry, and traceability to raw sources.Task-level cost or completion benefits remain unmeasured, so comparisons require paired runs sized to resolve the smallest decision-relevant change.Source: §14.3 · p. 185

14.4 Make standing context justify its permanent cost

Across 288 runs, Khatri (2026) found no measurable correctness improvement from context files, bounding effects to roughly 10 to 15 percentage points; Gloaguen et al. (2026) reported roughly 20 percent higher inference cost.These null findings support a no-file baseline but remain limited to the studied agents, models, repositories, tasks, and equivalence bounds.Source: §14.4 · p. 185
Standing context should retain only unusual constraints and failure-prevention rules tied to observed failures or measured outcomes, while common summaries remain available through on-demand retrieval.Context files should be maintained like configuration, with owners, review triggers, deletion conditions, and mechanical checks for counts, paths, commands, and source drift.Source: §14.4 · pp. 186–188

14.5 Set the context operating limits

Context operating limits should be set from production-shaped workloads by measuring correctness, reliability, and cost across context sizes, then placing the harness limit below the weakest task’s measured decline.Paired no-file comparisons and category-level ablations determine whether standing instructions justify permanent cost and which instructions remain.Source: §14.5 · pp. 188–189

Sources and evidence

The cited evidence spans strong, directional, and null or conflicting records, including long-context benchmarks, multi-turn failure studies, and empirical analyses of repository context files.The evidence record identifies Gloaguen et al. (2026), Khatri (2026), and Chatlatanagulchai et al. (2025) as relevant context-file studies.Source: Sources and evidence · pp. 189–190

Cross-session memory, raw traces, and compaction policies

Cross-session memory should preserve append-only raw events and provenance while rebuilding derived layers, because repeated rewriting can accumulate deletion and distortion.The complete chronological record of inputs, actions, outputs, and state changes is the rebuildable source record; added complexity should follow observed query and task failures.Source: Cross-session memory, raw traces, and… · p. 191
The evidence supports memory architectures and measurement decisions but provides no defensible retention period, retrieval threshold, or compression ratio for the contested storage recommendation.Seven supporting items include only one strong item, four syntheses, and one practitioner account; no controlled comparison of storage architectures is available.Source: Cross-session memory, raw traces, and… · p. 191

15.1 Keep the source record rebuildable

Keeping immutable raw events separate from derived summaries makes omissions repairable from preserved evidence rather than from already reduced model-generated inferences.The recorder preserves original session events, while a distiller creates purpose-specific summaries or extracted facts; without the trace, later repair can only infer over reduced evidence.Source: §15.1 · pp. 192–193
Preserving source events also retains temporal succession and supports derived representations of validity intervals, uncertainty, or historical status.Continuous rewriting can erase earlier states, while preserved events allow the derived layer to represent changes such as ownership succession.Source: §15.1 · p. 193
The architecture must treat raw traces as sensitive, governed source material because retention, deletion, and derived copies create privacy, access, and stale-claim risks.Deletion must propagate through summaries, embeddings, caches, and graph edges, while retention may require redaction, shorter windows, deliberate noncollection, or irreversible deletion.Source: §15.1 · p. 194

15.2 Add memory infrastructure only for measured retrieval failures

A simpler relational or full-text substrate should be the starting presumption, with vector or graph infrastructure added only after measured lexical or relational retrieval failures.The evidence does not establish relational storage as best across workloads; vector indexes address recurring lexical mismatch, whereas graphs address recurring relational traversal and impose additional maintenance obligations.Source: §15.2 · pp. 195–197

15.3 Let observed failures determine what compaction preserves

26 to 54 percent peak-token reductions and up to 46 percent gains for small-model agents accompanied improved task success in Kang et al. (2025), but coding-agent transfer remains an inference.The study covered app, office, and question-answering benchmarks rather than coding agents.Source: §15.3 · p. 200
Compaction policies should be revised from failure traces that identify omitted, distorted, or distracting evidence, then tested with paired runs for correctness rather than token reduction alone.The failure corpus should distinguish damaged context from downstream misuse and preserve raw traces for replay and rebuild.Source: §15.3 · pp. 200–201

15.2 Add memory infrastructure only for measured retrieval failures

Memory infrastructure should preserve permitted raw traces separately from derived summaries, embeddings, and graph edges, with explicit provenance, versioning, retention controls, and rebuild paths.Derived stores may be regenerated from authorized source segments, so they should not replace the independent source record.Source: §15.2 · p. 198, §15.4 · p. 201

15.4 Build memory in reversible stages

Retrieval should begin with a relational event store and full-text search, adding vector or graph components only for measured query failures while recording cost, provenance, deletion, and access-control consequences.A graph or vector lane can become a maintenance burden when introduced for anticipated questions rather than observed workload needs.Source: §15.2 · p. 198, §15.4 · pp. 201–202

Sources and evidence

The evidence record distinguishes strong support for failure-driven compression from directional support for light stores and preserves companion records without treating their claims as developed chapter practices.Kang, M., et al. (2025) supplies strong evidence for compression comparisons, whereas memory-store sources are identified as directional or anecdotal.Source: Sources and evidence · p. 203

Efficient verification interfaces and risk-based human escalation

Verification must be cheaper than uncritical acceptance; Vasconcelos et al. (2023) found overreliance changed when checking effort or error salience changed, while explanations were not a general antidote to misplaced trust.Review capacity remains finite even as generation scales; InfoQ reported that Stripe’s Minions produced more than 1,300 production pull requests weekly.Source: Efficient verification interfaces and… · pp. 205–206
Review controls must account for autonomy, duration, action space, and workload because evidence from suggestion-level or isolated-output review does not settle behavior for long-horizon agents.Older oversight findings remain useful for identifying mechanisms, but transfer to autonomous coding actions is rarely tested.Source: Efficient verification interfaces and… · p. 206

16.1 Make correctness cheap to challenge

Verification interfaces should expose executable evidence that lets reviewers discover errors, rather than merely explain why an output might be correct.Tests, type checks, replay, sandboxes, and evidence beside claims can make failures challengeable, but passing checks remain limited to their tested conditions and scope.Source: §16.1 · pp. 206–208
A large diff with weak tests or unsound scoping makes verification more expensive than acceptance, encouraging shallow engagement and producing artificially greener monitoring.The design objective is to automate checks where possible while making remaining uncertainty visible; explanations without practical checks are context, not evidence of safety.Source: §16.1 · pp. 208–209

16.2 Interrupt acceptance where overreliance occurs

Cognitive forcing functions reduced overreliance without eliminating it, and their effectiveness imposed usability costs that differed across participants.Buçinca, Malaya, and Gajos (2021) tested 199 participants; benefits accrued disproportionately to people with high Need for Cognition, while effective friction was rated least favorably.Source: §16.2 · p. 209
Review friction should target the decision context rather than apply uniformly, because acceleration involves shallow validation whereas exploration already includes comparison and rejection.Barke et al. (2023) studied suggestion-level interaction among twenty programmers, so this distinction remains a hypothesis for autonomous-agent review rather than a validated classification.Source: §16.2 · pp. 209–210

16.3 Allocate review where it can change the outcome

Approximately 15 percent higher true-positive rate at a 0.01 false-positive rate was reported for targeted escalation over blanket oversight, although the result came from a covert-misbehavior arena rather than production software review.Kale et al. (2025) also found that hierarchical and sequential monitor scaffolds allowed weaker models to oversee stronger agents; the operating point may not persist at another threshold.Source: §16.3 · p. 211
Human escalation is effective only when each flag identifies a decision owner, supporting evidence, a verdict, paused execution state, and resumption event.Otherwise, a flag is merely a notification that can remain unread, circulate among teams, or suspend execution indefinitely.Source: §16.3 · p. 212

16.4 Build the review and escalation path

Review paths should attach falsifiable artifacts to acceptance claims, force independent judgment before revealing recommendations, and measure detection, delay, bypasses, and workarounds rather than satisfaction alone.Strong evidence finds that explanations increase engagement when verification is cheaper or stakes are higher, while cognitive forcing functions reduce overreliance where explanations fail.Source: §16.4 · p. 215, Sources and evidence · pp. 216–217

16.3 Allocate review where it can change the outcome

Review should route exact violations to deterministic rules, contextual cases to versioned model judges, sequence-dependent cases to agent judges, and final verdicts to named human owners.Each layer must expose evidence to the next; the ordering is a design convention rather than an independently measured result.Source: §16.3 · p. 212, §16.4 · p. 216

16.4 Build the review and escalation path

Escalation must be budgeted around expected consequence, reversibility, novelty, disagreement, evidence quality, and qualified ownership, with explicit pause, resumption, and stale-approval rules.The review queue must translate operating-point rates into expected human-case volume, because low false-positive rates can still overwhelm review when benign actions dominate.Source: §16.3 · pp. 213–214, §16.4 · p. 216

Sources and evidence

Approximately 15% true-positive-rate gain at FPR=0.01 came from targeted escalation in one covert-misbehavior comparison at one operating point.Kale et al. report that scaffolding mattered more than monitor awareness; the setting was a covert-misbehavior arena rather than production code review.Source: §16.3 · pp. 213–214, Sources and evidence · p. 217

Autonomy calibration, provenance, effective gates, and accountability

Approval gates provide only assent unless execution enforces them: an audit found fail-open scripts and unvalidated command paths that allowed actions to bypass a human-only boundary.The evidence base for these accountability practices comprises 2 strong, 5 directional, and 0 corroborating items across four developed practices, so they remain testable defaults rather than universal policy.Source: Autonomy calibration, provenance,… · p. 219
Autonomy, provenance, gates, and accountability are effective only when operational mechanisms enforce the boundaries, information, control, and authority their policies describe.Explicit transfer records identify the initiator, proposed action, approving or executing party, supporting artifact, and recorded outcome, enabling calibrated autonomy and assignable accountability.Source: Autonomy calibration, provenance,… · pp. 219–220

17.1 Widen authority one action class at a time

Authority should widen one action class at a time, using class-specific outcomes, complete transfer records, uncertainty estimates, reviewer comprehension, and observed evidence rather than aggregate approval rates.Mixing action classes, excluding rejected or timed-out proposals, or recording only executed actions creates misleading rates; rare high-consequence failures may prevent wider autonomy despite clean observations.Source: §17.1 · pp. 220–222

17.2 Put provenance where review happens

Provenance labels improve review only when placed at the review surface and connected to verification, because Tang et al. (2024) found disclosure increased searching, verification, and repair while also increasing cognitive workload.Persistent provenance supports attribution and incident reconstruction, but its effectiveness depends on reviewer behavior, available tools, queue constraints, and whether the label survives repository workflows.Source: §17.2 · pp. 223, 225

17.3 Prove that a gate can change execution

A human gate is credible only when approval, rejection, or modification causally changes the next reachable execution state; otherwise it is compliance theater, and its effectiveness remains unresolved without deployment-specific testing.The cited oversight sources justify an audit structure and burden of proof but do not validate a universal test for effective gates.Source: §17.3 · p. 226, Figure 17.2 · p. 226
Approval queues and reproduction hooks failed when missing dependencies or command-construction flaws let actions bypass human control or made gates fail open.The cases show that installing a gate does not establish causal power, and the observed failures do not estimate general failure rates.Source: §17.3 · pp. 227–228

17.4 Assign responsibility only where control exists

Responsibility should be assigned only where a role combines authority, evidence access, intervention tools, and sufficient time to alter the relevant state transition.Organizations should repair the role by granting these controls under least privilege or move responsibility to the role that already holds them.Source: §17.4 · pp. 229–231

17.5 Audit the path from policy to execution

Audit policy-to-execution paths in order: define the action class, verify autonomy, test provenance persistence, establish gate binding, and then assign responsibility.The sequence prevents provenance and accountability judgments from preceding evidence that the action and gate are operationally defined.Source: §17.5 · p. 232

17.6 Test the autonomy transfer

Transfer autonomy one rung at a time only after recording all proposal outcomes, estimating approval and outcome uncertainty, and confirming reviewer understanding of state, evidence, and rollback.Rejected, modified, timed-out, abandoned, and executed proposals must remain distinct and action classes or autonomy rungs must not be combined.Source: §17.6 · p. 232

17.7 Test whether provenance survives review and integration

Test provenance markers through rebases, cherry-picks, squashes, file moves, copied patches, partial adoption, and human edits to preserve distinctions among generation, revision, and integration responsibility.The earlier 14.5 percent trailer-signed figure is only a lower bound on visible trailer-marked authorship, not an inferred contribution share.Source: §17.7 · p. 232

17.8 Test the gate as a failure experiment

Gate audits should inject dependency failures and alternate execution paths, then verify that rejection prevents every execution route and modified actions receive renewed review.The audit must also assess reviewer evidence, causal power, epistemic access, self-control, and fitting intentions in a contained environment.Source: §17.8 · p. 233

17.9 Audit accountability in two passes

Accountability auditing begins by selecting a real action and testing each accountable role’s deployed permissions rather than relying on administrative demonstrations.The supplied material introduces this as the first of two audit passes, with the role’s actual execution identity and control path as the basis for testing.Source: §17.9 · p. 233
The two-pass accountability audit tests whether a role can reject, inspect, stop, recover, and reverse actions, then compares those controls with users’ experienced expectations and execution traces.It records authority gaps, expectation mismatches, untraceable actions, failed stops, stale approvals, ineffective escalation, and late interventions, repeating passes when system or operating conditions change.Source: §17.9 · p. 233

Sources and evidence

Evidence is thin across the chapter: four developed practices carry seven evidence items, with two strong and five directional, while two accountability practices carry no strong evidence item.Recorded support includes strong studies linking behavior or verification to provenance disclosure, alongside directional oversight and meaningful-control frameworks whose operational transfer remains limited.Source: Sources and evidence · pp. 234–235

Part VI

Part VI marks the transition to a research agenda on work allocation.The supplied passage provides only this agenda heading fragment.Source: Part VI · p. 236

18.1 When does a persistent failure require structural repair?

A model upgrade or prompt change does not establish repair: persistent failures require paired repeated trials, localization of the earliest supported cause, and intervention at the boundary that produced it.Structural changes can repair failures that prompts leave intact, but they add handoff, latency, state, evidence, verifier, and context-loss risks.Source: §18.1 · pp. 237, 238, 240
Semantic search answered 65.2 percent of SWE-QA repository questions correctly versus 46.2 percent for planner-to-subagent search, at less than half the cost per correct answer, with handoff loss causing 41.8 percent of subagent failures.The comparison supports topology selection on SWE-QA, but evidence beyond read-only repository questions remains directional.Source: §18.1 · p. 238

18.2 When does fan-out beat the live single-agent baseline?

Coordination must clear both a success requirement and a cost limit against a live single-agent baseline; in SIMAS, multi-agent configurations used roughly 15 times as many tokens, with non-monotonic returns and debate sometimes underperforming self-correction.Aggregation is part of the treatment: an exploratory plurality-vote oracle gap reached 32.3 percentage points when voting selected an incorrect candidate despite a correct answer being available.Source: §18.2 · pp. 240–241
Fan-out evaluations must retain candidate-level results, aggregation decisions, and discarded oracle-correct candidates while measuring latency separately from end-to-end computation and cost.Parallel execution may reduce wall-clock time while increasing tokens and monetary cost, so the acceptable cost axis and ceiling should be declared before treatment results are known.Source: §18.2 · pp. 241–242
A multi-agent treatment counts only when traces show defining coordination behavior; otherwise configuration can masquerade as delegation while the system remains effectively single-agent.The benchmark treatment recorded zero subagent spawns across roughly 200 trials, showing non-entry into the causal path rather than subagent ineffectiveness.Source: §18.2 · p. 242

18.3 Which topology fits the task and contains its faults?

Closed-loop architectures neutralized more than 40 percent of synthetic faults that collapsed a linear pipeline, although stronger foundation models were not uniformly more robust.The comparison covered three MAS-FIRE architectures, so it does not establish a general preference for closed loops.Source: §18.3 · pp. 243–244
Topology must be tested through observed state transitions and fault routes, because shared corrupted premises, inaccurate summaries, and unqualified shared claims can turn local errors into global failures.Observation routes, provenance, disputed-state handling, and convergence points govern error correlation, recovery, and attribution even when workers share the same model and prompt.Source: §18.3 · p. 245

18.4 Which work is eligible to dispatch?

Dispatch should select the maximum independent set of nonconflicting work, transitioning nodes from blocked to ready only after prerequisites succeed and declared resources no longer conflict.Dynamic execution can create new repair or migration nodes, but semantic branching decisions still require a model or person to propose dependencies that the scheduler validates under policy.Source: §18.4 · pp. 247–248
No controlled result establishes that dynamic dependency graphs improve repository work relative to fixed schedules.DynTaskMAS compared parallel with serial execution on one seven-agent travel-planning workload, not dynamic scheduling with a well-designed fixed schedule [Yu et al. (2025)].Source: §18.4 · p. 248

18.5 A lease is not write authority

Safe reassignment requires ownership epochs at protected mutation boundaries, because leases permit reallocation but do not prevent stale workers from writing.The current ownership generation and attempt identity must be validated by the target or an authoritative mutation gateway [Burrows (2006)].Source: §18.5 · p. 250

18.6 What evidence should promote a topology?

Topology promotion requires mechanism entry, visible fault containment, and coordination return within predeclared success, reliability, latency, review-burden, and cost limits.Baseline and treatment must use identical task versions and preserve complete configuration identity; arrangements such as “hierarchical” or “multi-agent” do not establish adoption value.Source: §18.6 · pp. 252–253

19.1 What allocation decision is the system making?

Cheapest-sufficient routing selects the least expensive worker predicted to satisfy each request’s declared requirement, while deadline shipping returns the best feasible incumbent.The worker may be a model tier, model-plus-tool configuration, or execution pool; feasibility includes capacity, hard deadlines, and in-flight work.Source: §19.1 · p. 256
Allocation must define sufficient performance, computation deadlines, inviolable constraints, running-work representation, reassignment cost, and the uncertainty measure retained for operational decisions.The optimality gap distinguishes a plan known to be close to optimal from a merely feasible plan found before search progressed, even though both may be executable.Source: §19.1 · pp. 257–258

19.2 When should the scheduler recompute, and when should it stop?

A scheduler should recompute after important state invalidation or when recomputation costs less than repairing an obsolete plan, while retaining the incumbent, solve time, and optimality gap under a deadline.Running work should normally be represented as fixed commitments, and the objective should charge for churn; cadence and deadlines remain deployment decisions requiring pilots and comparison with a tuned greedy baseline.Source: §19.2 · pp. 258, 260

19.3 Admission, backpressure, and recovery capacity

In replay, overload-policy effects concentrated in one contended pool, while the uncontended pool showed almost no policy difference; recovery experiments should therefore compare bounded and unrestricted traffic under workload-specific capacity.The proposed experiment measures live-work tail latency, recovery completion, queue depth, retries, dependency request rate, starvation, throughput, and duplicate effects; no universal recovery-capacity fraction is prescribed.Source: §19.3 · pp. 260, 263

19.4 Which worker is cheapest and sufficient?

159 of 266 SWE-bench Pro tasks were solved at about one fifth the reported cost per solve, but the no-router ablation tied the routed system, locating the gain in verified handoff rather than routing.This result supports the benchmark-specific scout-and-route configuration; generalized cheapest-sufficient routing remains directional and depends on calibrated performance predictions.Source: §19.4 · p. 264
Confidence-gated cascades often outperformed always-on strong-model fan-out in both cost and quality, but each cascade must be evaluated as one complete route.The evidence supports the catalog pointer rather than a universal deployment rule because escalation changes latency and couples stages through the confidence signal.Source: §19.4 · p. 266
Cheapest-sufficient routing cannot repair unrepresentative calibration data, incorrect rewards, or costs that appear only after repair and review.Benchmark-derived routing may omit production task distributions, model-pool changes, repair costs, review costs, or realistic prompt formats; learned policies should therefore retain static fallbacks and explicit uncertainty.Source: §19.4 · pp. 265–266

19.5 Under what serving conditions should that inference run?

Prefix caching in a radix tree produced up to 6.4 times higher throughput on workloads with shared structure, including agent control and multi-turn chat.Whether an agent workload has reusable structure depends on the harness, prompt construction, and serving implementation; the cited serving work was not measured on a coding-agent fleet.Source: §19.5 · pp. 267–268
Serving saturation can create apparent agent failures through queueing, timeouts, retries, cache eviction, and retry-amplified demand without any reduction in model capability.Operators should bound admitted demand, cap and jitter retries, reserve recovery capacity, and attribute the first upstream failure rather than stopping at an agent timeout.Source: §19.5 · p. 268

19.6 Does the policy survive a fixed-arrival replay?

Across eleven weeks, a four-feature weighted index performed within 0.1 percent of plain priority banding, while P1 tail latency moved from 20.4 hours to 18.8 hours without triggering starvation.The improvement concentrated in one contended pool; an uncontended pool showed almost no difference across policies, and the replay remains a reasoned transfer rather than a controlled agent-fleet result.Source: §19.6 · pp. 268–270
Fixed-arrival replay enables paired policy comparisons only when the trace preserves feasibility context, including arrivals, eligibility, resource state, outcomes, and work already in flight.The harness should first reproduce known schedules, then compare cheap baselines under a predeclared objective, report outcomes by class, and guard against starvation.Source: §19.6 · pp. 269–270
Fixed-arrival replay should hold demand constant, compare candidates against cheap baselines, check per-class starvation and constrained capacity, and require shadow or canary evidence before rollout.Historical replay can miss policy-induced workload changes, so temporal holdouts and staged deployment are required; recovery experiments should measure latency, queues, retries, dependency load, starvation, throughput, and duplicate effects.Source: §19.6 · pp. 271–272

19.7 What must be recorded before optimization?

Before optimization, the ledger must reconstruct arrivals, eligibility, resources, alternatives, routing reasons, costs, reviewed outcomes, and later events, while preserving immutable versioned decision and estimate records.Observed events should be recorded before policy estimates; without shared decision identities and timestamps, analysis cannot distinguish worker performance from selection policy or establish counterfactual effects.Source: §19.7 · pp. 273–274

Sources and evidence

The supporting evidence remains directional: observatory scheduling, cluster systems, serving benchmarks, and practitioner guidance provide mechanisms, but no cited source establishes a production software-agent fleet policy or default recovery-capacity fraction.The record explicitly distinguishes strong evidence for particular benchmark or production-characterization claims from transfer decisions that still require fleet measurement.Source: Sources and evidence · p. 274, Admission, backpressure, and recovery… · pp. 274–275

Admission, backpressure, and recovery capacity

Cheapest-sufficient routing and bounded recovery are developed as practices to test, not established universal gains, because routing evidence is limited and recovery actions can themselves create severe overload.Serving optimizations change the resource envelope, but their reported gains depend on evaluated systems, models, workload structure, and implementation rather than proving a general routing policy.Source: Admission, backpressure, and recovery… · p. 275

agents

Across 370 paired tasks, retrieval measures improved repository-evidence placement while end-to-end reward rose only +0.0349, and the initial uncertainty estimate was invalid because tasks clustered within 46 repositories and 20 suites.The retrieval improvement was real, but its downstream uncertainty required dependency-aware resampling rather than treating tasks as independent.Source: agents · p. 277

20.1 Each layer determines what the next may trust

A grading system is an instrument that yields a verdict under stated conditions, while incomplete records, stale retrieval, and weak attribution can make downstream explanations or reliability claims invalid.The strongest automated attribution method located the decisive step in only 14.2 percent of 127 expert-annotated multi-agent failure cases, and stale context can remain fluent while losing authority.Source: §20.1 · p. 278

20.2 Engineering controls can precede prevalence evidence

Evidence strength is uneven and does not track operational consequence: strong-graded items comprise 56 percent of Part I evidence but only 10 percent of Part III, while Chapter 8 has none despite potentially urgent containment needs.This supports engineering controls before prevalence evidence when production write access makes containment operationally important.Source: §20.2 · p. 279
Evidence strength and operational urgency were effectively unrelated in a 52-practice subset, so controls may precede prevalence evidence when their mechanisms are directly testable.Such controls establish local protection and falsifiable observations, not universal prevalence or effect sizes.Source: §20.2 · pp. 279–280

20.3 Start smaller than this book

A small team can retain four essentials: executable acceptance checks, separated ordinary and recovery authority, versioned state identity, and paired comparison against a credible baseline.These controls show whether work passed, reconstruct the accepted artifact, prevent silent authority retention, and test whether added complexity helps.Source: §20.3 · p. 282

20.4 Some uncertainty should remain visible

Universal settings for repeats, agreement, context, retrieval, retries, recovery, grading, and scheduling remain unsettled, while many empirical conclusions are dated snapshots.Structural constraints on reasoning about systems are expected to outlast current models, runtimes, benchmarks, and costs.Source: §20.4 · pp. 282–283

20.5 The system around the model

Coding-agent outcomes depend on the surrounding system as well as the model, requiring a dependency-ordered debugging process before adding machinery.The minimum pass checks repeated results, verdict calibration, durable reconstruction, evidence freshness and delivery, and consequential boundaries under failure.Source: §20.5 · p. 283

Sources and evidence

The evidence appendix introduces no new evidence; its catalog shares are recomputed from the companion catalog, with Part I strongest and Chapter 8 lacking strong or direct scholarly evidence.Reported strong groupings are Part I 18/32, Part III 3/30, Part IV 20/42, and Part VI 8/29.Source: Sources and evidence · pp. 283–284

Practice catalog index

The practice index resolves 206 reliability records—193 gated practices, including 56 developed in the main chapters, plus 13 research leads—with stable identifiers and treatment labels.The index records placement and identity, while the companion catalog supplies each record’s action, mechanism, evidence, and boundary.Source: Practice catalog index · p. 285, Table A.1 · p. 285
The catalog prioritizes human-auditable traces, constraint-violation logs, hierarchical context, and independent judges for failure attribution.It also recommends distinguishing agent from environment failures and fixing recurring failure classes structurally rather than with prompt patches.Source: Practice catalog index · p. 287
Retrieval and context practices treat access as an iterative, separately scored process governed by relevance, freshness, authorization, structure, and measured context limits.The index includes hybrid retrieval, staged localization, AST-aware chunking, deterministic anchors, and persistence of explored context across reasoning steps.Source: Practice catalog index · pp. 287–288
Evaluation practices require execution-grounded checks, calibrated judges, live-data measurement, replayable golden sets, and separation of submission from verified resolution.The index also calls for external feedback before self-correction, per-task consistency signals, and evaluation on the system’s own workload.Source: Practice catalog index · pp. 287–288
State and recovery practices call for durable workflow state, transactional and idempotent execution, bounded replay, measured checkpointing, fault-injection benchmarks, and evidence-grounded retries.They also cover persistent artifacts, provenance-driven re-execution, restore-point certification, and deterministic post-condition guards after stochastic generation.Source: Practice catalog index · pp. 288–289
Oversight practices require calibrated confidence, measured autonomy, effective verification surfaces, explicit control and oversight roles, and human escalation for monitor-flagged cases.The index rejects self-attestation, consensus as verification, and oversight without sufficient understanding or authority.Source: Practice catalog index · pp. 288–290
The practice catalog extends reliability guidance from human oversight through typed retrieval, scheduling, allocation, and recovery controls.Entries include real expert override authority, decomposed oversight metrics, typed knowledge-graph retrieval, audited allocation, and recovery protections against stale authority, retry storms, and starvation.Source: Practice catalog index · p. 290

Glossary

Calibration relates an instrument’s outputs to a reference outcome under stated conditions, so agreement is a result of calibration rather than correctness itself.The glossary also distinguishes inter-rater reliability as reproducibility among reviewers and defines operating points through threshold, workload, prevalence, and cost conditions.Source: Glossary · pp. 292–293

Data and materials availability

The manuscript source and companion artifact publicly provide the 206-record catalog, evidence ledger, benchmark materials, schemas, protocols, skills, evidence maps, and checksums.The repository also provides pinned compilation records and public CodeScaleBench materials, while workflows are identified as implementation artifacts rather than independent evidence.Source: Data and materials availability · p. 313
The 199-trace diagnostic corpus and 1,286-item fleet ledger are not redistributed because de-identification would remove information needed to audit causal and dependence structure.Their aggregate findings are therefore identified as author-system illustrations rather than independently reproducible external evidence.Source: Data and materials availability · p. 313