PaperlayerView exact revision on arXiv

Source-linked AI summary

Audit the Scaffold, Not the Checkpoint: A Stationarity Dichotomy for Recursive Self-Improvement in Agentic Coding

Sebastian Bobadilla-Suarez, Bob Suh, Ryan Fortin

arXiv:2609.34924v1cs.LGcs.AIcs.SE

TL;DR

The paper asks when iterative self-modification saturates and whether frozen weights establish that regime. It formalizes refinement through boosting and studies reachable edit classes, orchestration, and worker overlap. It concludes that fixed reachable sets force diminishing returns, while scaffold expansion can escape only under a sustained edge and within the model’s limits.

  • Problem

    The paper addresses the lack of a rigorous foundation for when iterative self-modification runs out of room and whether frozen weights establish that limit.

    Source: §1 · p. 1

  • Method

    It models coding refinement as residual-directed patching and analyzes reachable edit classes, orchestration, and worker overlap using boosting-based formal results.

    Source: §1 · p. 2, §4 · pp. 4–5

  • Results

    Fixed reachable edit sets force strict diminishing returns, while escape requires expansion; on the measured worker pool, m⋆=k and the majority fails 23/55 (42%) of tasks.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    Auditing scaffold expansion rather than frozen weights tracks whether the reachable edit set can grow, though expansion alone does not ensure escape.

    Source: §4 · p. 5, §6 · p. 8

    The production dynamics are consistent with bounded behavior but do not establish that a fixed hypothesis class causes it, and the reported edge estimates condition on tasks still live.

    Source: §4 · p. 5, §6.1 · p. 8

Abstract

from arXiv · show

An auditor who checks whether a system's weights are frozen is checking the wrong thing. Our stationarity dichotomy says that iterative self-modification hits strict diminishing returns whenever the agent's reachable set of edits stays fixed, and can escape only if that set expands. Rewriting scaffolding (tools, verifiers, decomposition) expands what an agent reaches without touching a weight, so frozen weights buy an eventual ceiling but no stationarity along the way. The criterion also separates three regimes usually merged: search within a fixed class, test-time training that raises the ceiling itself, and scaffold rewriting between them. Audit the scaffold, not the checkpoint. The same ceiling binds sideways. Best-of-$k$ orchestration realizes the best worker's ceiling exactly: width buys rate, not budget. Re-consulting a fixed pool has a horizon computable in advance, decided by the pool alone, and the one arrangement that would beat it, a weighted vote, needs diversity real workers lack: on 30 same-family workers the failure overlap sits at its maximum, and a majority fails 23/55 (42%) of tasks. We obtain the criterion by reading refinement as gradient boosting on the residual error between draft and target, a patch or git diff, and then measuring where that reading breaks: patches compose instead of standing beside each other to be voted on, and failures overlap. What we measure is saturation. Per-round improvement decays toward zero on SWE-bench, and churn decays geometrically across 401 production sessions, a shape shared with a pre-AI human baseline that establishes the regime without identifying its cause. Both breaks are engineering choices rather than laws about code, so together they specify a harness worth building.

1 Introduction

The paper asks when iterative self-modification saturates and whether frozen weights establish that regime. It proposes a reachable-edit-set criterion and examines where the boosting analogy fits deployed coding loops.

1 Introduction

  • Agentic coding uses multiple agents to generate, critique, and refine code, but its choices about team size, patch formulation, and stopping remain largely ad hoc.The paper frames it as a microcosm for recursive self-improvement [Good, 1965, Bostrom, 2014].Source: §1 · p. 1
  • A fixed reachable set of edits implies strict diminishing returns; escape requires continued expansion of that set.Scaffold rewrites can expand the reachable set with frozen weights, so frozen weights alone do not imply stationarity along the way.Source: §1 · p. 2
  • The paper measures saturation on SWE-bench and production sessions, then uses failures of boosting preconditions to specify an unbuilt harness.The production dynamics are consistent with the bounded regime but do not establish that a fixed hypothesis class causes them.Source: §1 · p. 2

2 Related Work

The paper situates its framework in boosting, recursive self-improvement, and agentic coding, while distinguishing scaffold rewriting from search within a fixed class and from test-time training. It also separates verifier-based sampling coverage from the error of a verifier-free vote.

2 Related Work

  • The paper reads refinement as boosting on residual error and relates its structural split to bounded versus open-ended recursive self-improvement.Boosting as functional gradient descent motivates the residual-as-patch mapping [Mason et al., 1999, Friedman, 2001].Source: §2 · p. 2
  • Repeated sampling measures oracle-verifier coverage, whereas Proposition 6 concerns majority-vote error without such a verifier; growing coverage and a failing vote can coexist.Cross-vendor aggregators report gains but not the overlap statistic the proposition identifies as decisive [Wang et al., 2024, Jiang et al., 2023].Source: §2 · pp. 2–3
  • Inference compute searches within a fixed class, test-time training raises the ceiling, and frozen-weight scaffold rewriting occupies a distinct regime.A frozen-weight audit can miss scaffold expansion while detecting test-time training.Source: §2 · p. 3

3 Framework: Agentic Coding as Boosting

The framework treats a coding agent’s edit as a residual-directed patch, linking refinement to boosting while distinguishing sequential code composition from an additive weighted vote. Deployed loops lack key boosting preconditions, which the paper turns into a proposed harness specification.

3 Framework: Agentic Coding as Boosting

  • A weak coding agent predicts the residual between a draft and target through a patch, rather than generating the full file from scratch.A positive improvement edge in code space already presupposes non-trivial program understanding because random edits tend to break working programs.Source: §3 · p. 3
  • Code patches compose sequentially and order-dependently, while boosting combines learners through an additive confidence margin.The exponential-loss result applies to the margin, not to code bytes.Source: §3 · p. 3
  • Deployed loops overwrite drafts rather than retain proposals, and worker failures overlap, breaking two preconditions for the boosting guarantees.The measured overlap reaches its ceiling, m⋆=k.Source: §3 · p. 3
  • The proposed harness would retain all proposals, reweight specifications still failing, and provide enough diversity for majority failures below one half.These are engineering choices, and the paper does not build this harness.Source: §3 · p. 3
  • The framing identifies the reachable class and its ceiling as quantities to track, and motivates a harness specification beyond the criterion alone.It also supplies vocabulary for stating the structural question of whether a system can run away.Source: §3 · p. 3

4 Theoretical Analysis

The analysis formalizes refinement and derives a stationarity dichotomy: a bounded quality scale forces saturation, while sustained edge in an expanding class permits divergence. It then treats orchestration, voting, and repeated use of a fixed pool, with explicit modeling and empirical scope limits.

4 Theoretical Analysis

  • Lean checks the algebra but not the key assumptions or the mapping from coding systems to the formal model.The authors identify M1, M2, and the mapping itself as modeling assumptions that no proof check can establish.Source: §4 · pp. 3–4
  • The weak coding agent maps a draft and edit to a refined draft, with success defined as matching the target; its edge measures solving success.The setup uses a distribution over specifications and code targets and a loss on code artifacts.Source: §4 · p. 4
  • A fixed quality ceiling forces cumulative improvement to saturate and per-round gains to diminish; sustained uniform edge in an expanding class permits divergence.The result is a criterion, not a convergence rate.Source: §4 · p. 4
  • Frozen weights guarantee only an eventual ceiling under assumptions M1 and M2; scaffold expansion is necessary but not sufficient for escape.The escape clause requires a uniform positive edge, and both assumptions are modeling assumptions rather than theorems.Source: §4 · pp. 4–5
  • Best-of-k orchestration attains the best worker’s ceiling exactly, so width can accelerate improvement but cannot raise the total improvement budget.The result uses monotonicity of the inner action-expectation; hard selection has no strict regime, including for disjoint singleton classes with equal values.Source: §4 · p. 5
  • A majority vote has zero training error exactly when every task has fewer than k/2 failing workers; the condition is sharp.A single task at the threshold is enough for nonzero vote error.Source: §4 · p. 5
  • On the same-family pool, m⋆=k and the majority fails 23/55 (42%) of tasks, while the vote-error gap over mean worker error is inconclusive.The pool has one vendor and limited effective diversity, so it settles the pessimistic case of resampling one model, not whether different model families clear the overlap condition.Source: §4 · p. 5, Figure 1 · p. 6
  • Re-consulting a fixed pool has a finite computable horizon when no nonnegative combination fits the work; an interpolating pool sustains the edge across rounds.The horizon is bounded by log(1/ϵ0)/2γ^2 and is uniform over schedules; the interpolating case has ϵ0=0.Source: §4 · pp. 5–6

5 Experiments

The experiments measure refinement saturation on SWE-bench Lite and in production sessions, while distinguishing capability effects from feedback-mode effects and descriptive patterns from causal mechanisms.

5.1 Part B: LLM Agent Refinement on SWE-bench Lite

  • The experiment uses a four-round refinement loop across 55 SWE-bench Lite tasks and five seeds, varying model tier and feedback mode in 1,650 trajectories.Feedback was independent, prior-patch-only, or diagnostic; no arm was shown the gold patch.Source: §5.1 · p. 6
  • Model tier explains roughly half the variance (F(1, 54)=53.46, p<0.0001, partial η2=0.497), while feedback mode and its interaction are underpowered null results.Solve rates were 44–49% for Haiku and 85–89% for Sonnet at the study’s τ=0.6 criterion; these are not SWE-bench resolve rates.Source: §5.1 · p. 7
  • Mean per-round improvement on the running best falls 0.097 → 0.036 → 0.033 for Haiku and 0.156 → 0.068 → 0.029 for Sonnet, shrinking toward zero.The authors identify this as measured saturation; conditioning on still-live tasks shrinks and hardens the population over rounds.Source: §5.1 · p. 7

5.2 Part C: Refinement Dynamics in Production Agentic Coding

  • Production analysis covers 401 sessions and uses structural git metrics without retaining code content, file paths, commit messages, or author identities.Sessions comprise consecutive AI-authored commits with inter-commit gaps below four hours.Source: §5.2 · pp. 7–8
  • 70.0% of multi-commit sessions show decreasing churn, with a median late/early churn ratio of 0.49 (95% CI [0.400, 0.793]).The temporal ordering is significant in the 70 sessions with at least five commits (permutation test, p<0.0001); the 401-session evidence is descriptive, not mechanistic.Source: §5.2 · pp. 7–8
  • A 9,395-session pre-AI human baseline shows the same churn-decay shape, while the Opus–Sonnet survivor-ratio comparison is a bounded null, not evidence of zero.The human geometric decay factor is ρhuman=0.86 versus AI-tier ρ≈0.77; the Opus–Sonnet gap is −0.003 (95% CI [−0.075, +0.069], p=0.93).Source: §5.2 · p. 8

6 Discussion

The discussion recommends investing in first attempts and capability while monitoring scaffold self-expansion rather than frozen weights. It also identifies unresolved tests and limits on interpreting the empirical edge and production comparisons.

6 Discussion

  • Spend on the first attempt rather than the refinement loop, and prioritize capability over orchestration, which cannot exceed the best worker’s ceiling.The measured capability gap dominated varied factors (partial η2=0.497).Source: §6 · p. 8
  • Monitor whether the scaffold is self-expanding: rewriting tools or decomposition escapes the fixed-scaffold bound and may create reward-hacking risk if systems can rewrite their own tests.The paper’s governance framing distinguishes this from simply checking whether weights are frozen.Source: §6 · p. 8
  • The proposed tests include building a retained, reweighted, diverse harness, measuring cross-family vote failure overlap, continuing refinement after success, and varying scaffolds with weights fixed.The last design would distinguish a fixed reachable class from ordinary editing economics; Part C does not run it.Source: §6 · p. 8

6.1 Limitations

  • The data provide no clean unconditional per-round edge estimate because the harness stops calling the model once a task passes; later-round cells were never run.Reported estimates condition on tasks still live, whose population shrinks and hardens over rounds.Source: §6.1 · p. 8
  • Part C’s model tier is not randomly assigned, its survivor ratio is a structural proxy for quality, and its human contrast compares different eras.The Opus–Sonnet null is bounded rather than a demonstrated zero, so it does not establish absence of a capability effect.Source: §6.1 · p. 9

7 Conclusion

The paper frames recursive self-improvement as a question about reachable edits: fixed reachable classes saturate, while escaping that ceiling requires expansion. Empirical saturation and failures of the boosting analogy point to scaffold and harness design as practical concerns.

7 Conclusion

  • A fixed reachable class with a bounded metric forces strict diminishing returns; frozen weights alone do not guarantee a stationary trajectory.Rewriting the scaffold can expand the reachable set without changing weights, but expansion is necessary, not sufficient, for escape.Source: §7 · p. 9
  • Best-of-k orchestration reaches the best worker’s ceiling exactly, while a majority fails 23/55 (42%) of tasks and a fixed pool has a computable horizon.The weighted vote that could beat the ceiling requires diversity that the observed same-family workers lack.Source: §7 · p. 9
  • Refinement is not boosting in the technical sense because patches compose instead of standing alongside one another for voting.The paper treats this and overlapping failures as engineering properties of current harnesses, not laws about code.Source: §7 · p. 9
  • Per-round improvement decays toward zero on SWE-bench, and churn decays across 401 production sessions in a shape also seen in a pre-AI human baseline.The shared shape establishes a bounded descriptive regime without identifying its mechanism.Source: §7 · p. 9
  • The production data do not distinguish a genuinely fixed reachable class from ordinary editing economics; varying the scaffold while holding weights fixed would test that distinction.The paper argues that the remaining gaps are harness engineering choices and specifies a harness worth building.Source: §7 · p. 9

Responsible Use Statement

The section frames recursive self-improvement as a saturation result, not an acceleration recipe, and connects agentic coding to boosting through residual refinement and sequential patch composition. It also states the scope of the transferred boosting results and the assumptions they require.

Responsible Use Statement

  • The paper presents its finding as a saturation-and-limits result, releases no scaffold-expansion technique, and notes that commit timing can remain linkable to repository holders.The released datasets retain structural commit timing for 71 people while omitting code content, paths, and messages.Source: Responsible Use Statement · p. 9

A The Boosting/Coding Mapping, and the Protocol

B Full Proofs of the Theoretical Results

  • Lean checks the algebraic proof steps, but does not certify the modeling assumptions connecting the abstract refinement game to real systems.Those assumptions remain explicit hypotheses and empirical claims.Source: B Full Proofs of the Theoretical Results · p. 14

B.1 Statements of the Transferred Classical Results

  • The transferred boosting results require a retained, reweighted ensemble combined by weighted vote; real LLM patch generators may have unknown VC-dimension, non-i.i.d. examples, and a changing hypothesis class.The paper relies on none of the transferred classical results.Source: §B.1 · pp. 14–15

A.1 Algorithm: Agentic Boosting Protocol

  • The Agentic Boosting Protocol reweights specifications after selecting the agent with lowest weighted error, and returns sequentially composed patches with an additive confidence margin.It stops when weighted error is at least 1/2.Source: §A.1 · p. 14, §B.1 · p. 15

B.2 Proofs

The proofs distinguish bounded improvement within a fixed reachable class from sustained improvement when that class expands. They also establish that best-of-k selection over fixed workers reaches, but does not exceed, the best worker’s ceiling.

B.2 Proofs

  • With a fixed reachable patch class and bounded quality, total improvement is bounded; sustaining a positive improvement indefinitely requires the reachable class to keep growing.The fixed-class condition yields a plateau, while the expanding-class case can exceed any fixed bound.Source: §B.2 · p. 18
  • The frozen-weight ceiling gives an eventual bound, not stationary behavior on the way there, because the scaffold can expand the reachable class while weights remain fixed.The Lean-checked saturation bound does not formalize the separate monotone-convergence step to the limit.Source: §B.2 · p. 18

C Orchestration, Voting, and the Diversity a Vote Needs

  • The orchestration discussion concerns selecting among frozen workers; orchestration cannot raise an individual worker’s ceiling, and the setup permits distinct prompts, tools, or specializations.The paper treats orchestration as part of the mutable scaffold.Source: C Orchestration, Voting, and the… · p. 18

C.1 Hard Selection Reaches the Best Worker, Exactly

  • Best-of-k orchestration reaches exactly the highest individual worker ceiling, so routing among fixed workers realizes existing capability rather than exceeding it.The equality holds without conditions on how the workers’ reachable classes overlap.Source: §C.1 · p. 19

C.2 Weighted Voting, and the Diversity It Requires

This section examines when weighted voting and pool reuse can improve performance, focusing on the roles of worker diversity, sequential edge assumptions, and a pool’s reachable cone. It derives a majority-failure bound under product-measure independence while noting that this modeling assumption is violated by real workers and identifying what remains unformalized.

C.2 Weighted Voting, and the Diversity It Requires

  • exp(-2γ^2k) bounds the training error of an AdaBoost weighted vote, eventually putting it below 1/2−γ when each worker’s current reweighted error is at most 1/2−γ.The overlap criterion is exact: the unweighted vote has zero training error when every task has fewer than k/2 worker failures; counting alone gives only the weaker bound 2 times average worker error.Source: §C.2 · pp. 19–21
  • Width buys rate, not budget: multi-round best-of-k orchestration inherits the best worker’s per-round improvement but reaches the same frozen-weight ceiling sooner.The total improvement budget remains fixed by the bounded metric, so adding workers does not enlarge it.Source: §C.2 · pp. 19–20
  • 23/55 (42%) of tasks had 2m_i ≥ k among 30 same-family workers, whose tight overlap was m⋆=30=k at every round.The vote’s zero-one error was 0.418 (95% CI [0.291, 0.545]); the pool was one vendor, two tiers, and three prompt variants, so it does not test whether genuinely different families satisfy the overlap condition. Within either tier, overlap remained at its ceiling, but the pooled vote’s apparent excess error over average worker error was a tier-pooling effect.Source: §C.2 · p. 21

C.3 Reusing a Pool: A Horizon, or an Interpolating Cone

  • A worker reused immediately at its optimal confidence has exactly zero edge on the next round, so consulting that same worker cannot sustain the weighted-vote edge hypothesis.This rules out immediate self-reuse, not every reused pool: another pool member may retain an edge against the resulting distribution.Source: §C.3 · pp. 22–23
  • A fixed pool whose nonnegative cone cannot fit the training data must lose the edge hypothesis by a finite, computable round; an interpolating cone instead guarantees an edge at every round.The horizon is uniform over schedules and confidences, while interpolation gives a fixed edge γ=θ/(2S) for every distribution. The interpolation result is per-round and does not formalize a complete multi-round run.Source: §C.3 · pp. 23–24

C.4 Are the Two Conditions Equivalent?

  • The reweighted-edge condition and bounded-overlap condition are independent: each can hold while the other fails, with both counterexamples using four points and three workers.One construction has reweighted errors 1/4, 1/6, and 2/5 but fails the overlap condition; the other has m⋆=1<k/2 but includes a worker with error 3/4.Source: §C.4 · p. 24

C.5 The Probabilistic (Condorcet) Route

  • (1−4γ^2)^m ≤ exp(−4γ^2m) bounds majority-vote error for 2m workers with independent error probabilities at most 1/2−γ.Source: §C.5 · p. 25
  • The Condorcet majority-failure bound retains constant e−2γ2k, with worker-failure independence derived from a product Bernoulli measure and the event identified with Prop. 23’s voteErr.The bound is transported from Prop. 23 rather than reproved with Hoeffding’s inequality; the product-measure assumption itself is not established by the proof.Source: §C.5 · p. 25

C.6 Status: What Is Settled, and What Is Not

  • Reconsulting a fixed pool either interpolates the sample, yielding a member with an edge against every distribution, or fails the edge hypothesis by round log(1/ϵ0)/2γ2.Consulting the same worker twice zeroes its edge; the qualitative dichotomy follows from AdaBoost’s minimax duality, while the explicit horizon is the effective form established here.Source: §C.6 · p. 26
  • The reuse conditions m⋆<k/2 and the reweighted edge hypothesis are logically independent, and the concentration cap κ can be derived from logged confidences.Weighted aggregation can escape hard selection’s least-bad-worker floor only when individually mediocre workers satisfy the diversity encoded in the edge hypothesis; correlated or identical workers fail AdaBoost’s stopping rule.Source: §C.6 · pp. 25–26
  • The multi-round weighted-vote run is not formalized and requires excluding a perfect pool member; the product-measure assumption is contradicted by a 23/55 (42%) majority-failure rate.The Condorcet independence setup is a modeling choice, and the observed majority-failure rate shows real workers violate it.Source: §C.5 · p. 25, §C.6 · p. 26

D Part A: Additional Figures

Part A checks the boosting and generalization machinery on synthetic labels, while its figures show the expected diminishing returns, margin concentration, and test-error trends.

D Part A: Additional Figures

  • Part A is an implementation sanity check on synthetic labels, not a test of code, correctness, or LLM behavior.Its curves are comparable to Part B in shape, not in sign, because the parts use different edge estimands.Source: D Part A: Additional Figures · p. 26
  • Code quality rises with diminishing returns and remains below 1.0, while more capable agents converge higher.The refinement-game result uses Levenshtein-distance optimization.Source: Figure 2 · p. 26
  • Over 80 boosting rounds, test error for N = 500 decreases and then plateaus; final test error also falls as training-set size grows.The latter trend is averaged over five seeds on the full SWE-bench split with a disjoint held-out test set.Source: Figure 3 · p. 27
  • Across boosting rounds, negative-margin mass collapses and margins concentrate on the positive side of the boundary.The passage identifies this concentration away from the decision boundary as the mechanism for generalization despite adding weak learners.Source: Figure 4 · p. 27

E Part B: Harness Verification, Task Pool, and Power Analysis

Part B documents the evaluator and task pool used for its agent study, then reports a strong model-tier effect alongside feedback-mode results that are underpowered and inconclusive.

E Part B: Harness Verification, Task Pool, and Power Analysis

  • The evaluator was built for persistent per-round scoring instead of reusing SWE-bench’s batch evaluator, but the authors acknowledge reimplementing repository-specific commands and parsers.Agreement with the official grader on retained tasks was established by gold-patch checks, not side-by-side runs.Source: E Part B: Harness Verification, Task… · pp. 27–28
  • A gold-patch audit found 23/50 initially sampled tasks failing deterministically for evaluator reasons, prompting fixes, exclusions, and 15 verified replacements for a K=55 pool.The retained pool excludes Django, SymPy, and one network-dependent task; results use the fixed, gold-verified pool, including a pytest-native harness.Source: E Part B: Harness Verification, Task… · pp. 27–28
  • For final-round quality, the model effect in the 2 × 3 repeated-measures ANOVA (55 tasks × 5 seeds) had a Cohen’s f 95% CI of [0.738, 1.319], consistent with near-complete tier separation in solve rates.Source: Table 2 · p. 28
  • At τ≥0.6, solve rates are 44–49% for Haiku and 85–89% for Sonnet; final-round quality is threshold-free, unlike solve-rate tallies.The rate asymmetry is relevant because solved rows are back-filled at their running best, while some unsolved rows finish below an earlier score.Source: E Part B: Harness Verification, Task… · p. 28
  • The mode and interaction nulls are underpowered: at K=55, estimated power is 11% and 17%, and the sampling-noise floor exceeds the measured between-mode signal.The reported 80%-power sample targets are approximately 467 tasks for mode and 364 for interaction; uniform temperature 0.25 was chosen for simplicity although only independent requires it.Source: E Part B: Harness Verification, Task… · p. 29
  • The feedback-mode point-estimate crossover is not a confirmed finding: every formal test of its ordering is non-significant.Haiku's means increase across modes, while Sonnet's blind mean is its lowest; the passage rejects attributing that inversion to early solving followed by regression.Source: E Part B: Harness Verification, Task… · p. 29

E.1 Decomposing the Measured Edge

The measured edge is complicated by back-filled non-events and ties: pooled estimates can mislead, while eligible-cell and abstention-aware analyses clarify the observed saturation and its limits.

E.1 Decomposing the Measured Edge

  • The feedback-mode figure reports mean final-round quality across 55 tasks and five seeds, but Sonnet's blind-arm dip is a non-significant crossover.Haiku's mean quality increases monotonically across independent, blind, and diagnostic modes.Source: Figure 5 · p. 30
  • Back-filled solved rounds enter the pooled edge as failures to improve, so the stronger tier can appear to have a more negative edge despite stronger refinement on called cells.The pooled series is reported for disclosure rather than as a headline because it tracks solve rate backward.Source: §E.1 · pp. 29–30
  • Net of back-filling, genuine no-ops dominate transitions at 52–59% for Haiku and 13–21% for Sonnet, while quality regressions reach at most 2.9%.Across complete trajectories, 80.7% of Haiku rows and 69.1% of Sonnet rows never change quality; negative γ here indicates idempotence, not anti-correlation with correctness.Source: §E.1 · p. 30
  • Among rows still eligible to improve, Haiku's measured edge is negative in all three rounds, while Sonnet's first two intervals include zero and round 3 is negative.The eligible population shrinks as rows saturate, from 194 to 158 of 275 for Haiku and 113 to 49 of 275 for Sonnet.Source: §E.1 · p. 30, Figure 6 · p. 31
  • The eligible-cell correction is not an unconditional edge because it omits unrun post-pass rounds and selects against rows that did well earlier.The abstention-aware bound descends when unchanged rounds receive Zs < 1, whereas the binary bound stays at 1 because it cannot represent abstention.Source: §E.1 · p. 31

F Part C: Additional Findings and Figures

Part C finds geometric front-loading in AI-assisted and pre-AI sessions, but survivor-ratio comparisons do not establish a causal AI effect. Within-era comparisons are suggestive but weak, while the geometric model captures only the coarse shape of session outcomes.

F Part C: Additional Findings and Figures

  • The geometric churn-decay model captures the coarse survivor-ratio distribution but fails exact-fit tests, with held-out rejection varying across seeds.In-sample KS=0.21 (p<0.0001); held-out KS=0.18 (p=0.045), with rejection in 6/8 seeds, likely reflecting pooled repository and developer heterogeneity.Source: F Part C: Additional Findings and… · p. 32, Figure 8 · p. 37
  • Session lengths are heavy-tailed: median 1 commit, mean 3.0, and maximum 45, with long sessions driving much of the structural change.The log-normal reference is descriptive; a KS test rejects exact fit.Source: F Part C: Additional Findings and… · p. 32, Figure 9 · p. 38
  • The pre-AI baseline shares the geometric churn-decay pattern (ρhuman=0.86 versus AI-tier ρ≈0.77), but its survivor ratio is lower than both AI tiers, with era confounds precluding causal interpretation.The adjusted Opus-versus-human and Sonnet-versus-human coefficients are +0.145 (p=0.00082) and +0.154 (p=0.00026), respectively.Source: F Part C: Additional Findings and… · pp. 33–34
  • Among multi-commit sessions, mean non-pure survivor ratio is 0.435 for the control versus 0.593 for the AI tier, so the pooled gap is not explained by singleton-heavy controls.The cohorts differ by era and other factors, so this descriptive comparison supports no causal claim about AI.Source: F Part C: Additional Findings and… · pp. 33–34
  • Within the AI era, pooled mean non-pure survivor ratio is 0.540 for AI-assisted work versus 0.419 for non-AI work, but paired evidence is suggestive, not established.At the ≥50-commit floor, 5 of 7 developer pairs favor AI-assisted work (median ∆ψ=+0.127), but task selection is uncontrolled.Source: F Part C: Additional Findings and… · p. 34
  • The capability null is bounded: version-resolved rank tests find no trend, but their intervals still allow substantial effects, so they do not establish a zero capability effect.Family-ordering and version-ordering Spearman correlations are +0.012 (p=0.82) and −0.008 (p=0.88); intervals permit a swing comparable to the human–AI gap.Source: F Part C: Additional Findings and… · pp. 35–36

G.1 Related Work: Empirical Systems and Test-Time Compute

  • Related empirical work covers multi-agent coding, self-refinement, and test-time compute, while this paper positions its structural criterion as a distinct contribution.The cited literature reports diminishing returns for test-time compute; the paper describes its refinement-game prediction as first-principles theory.Source: §G.1 · p. 36

G.2 Could Decomposition Realize the Protocol Implicitly?

  • Refinement is boosting-inspired, not boosting-exact: patches compose rather than remain separate learners for a weighted vote, and decomposition lacks AdaBoost’s required update structure.Dynamic decomposition expands sub-specifications, while focusing on failing tests is not the multiplicative-exponential reweighting used in the transferred bound.Source: §G.2 · p. 37

G.3 Guidance for Coding in the Wild (full)

The guidance favors strong first attempts, early steering, and stopping once verification passes, while distinguishing added search from changes that expand an agent’s reachable set. The paper argues that auditing scaffold stationarity matters alongside weight changes, while noting that its empirical saturation evidence does not identify the mechanism causing it.

G.3 Guidance for Coding in the Wild (full)

  • Per-round improvement decays across refinement rounds, so investing in the first attempt is favored over extending the loop.Reported improvement rates fall from 0.097 to 0.036 to 0.033 and from 0.156 to 0.068 to 0.029; extra rounds are rarely harmful, making them wasteful rather than dangerous.Source: §G.3 · p. 38
  • The verifier is a proposed residual-error signal, but the K=55 ablation did not show diagnostic feedback significantly outperforming blind resampling or independent redraws.The feedback-mode effect was a small, underpowered null (Cohen’s f=0.10, 11% power); the study tests self-critique against test-failure feedback, not a separate reviewer model.Source: §G.3 · p. 38
  • The paper recommends capability before orchestration because orchestration can realize an existing ceiling sooner but cannot create a higher one.Part B’s stronger model wins by clearing more tasks on the first try, not by refining better.Source: §G.3 · pp. 38–39
  • Re-sampling and more rounds explore within a fixed class, whereas new tools, retrieval, tests, or decomposition can expand the reachable class and lift its ceiling.The paper says the theory does not predict a large gap among blind, independent, and diagnostic feedback when weights and tools stay fixed.Source: §G.3 · p. 39
  • Production churn is front-loaded, supporting early review and redirection; the guidance also recommends stopping when the verifier passes to avoid regressing a correct artifact.The reported churn-decay factor is ρ≈0.77; Algorithm 1 uses a deterministic margin gate.Source: §G.3 · p. 39

G.4 Implications for Recursive Self-Improvement (full)

  • A fixed reachable class forces strict diminishing returns, while escaping that regime requires continued class expansion under the stationarity dichotomy.The practical audit question is whether the set of reachable edits is stationary, not simply whether model weights are frozen.Source: §G.4 · p. 39
  • Test-time training changes weights and therefore the ceiling, while scaffold self-expansion can change reach without a weight update; both must be distinguished in evaluation and governance.The paper argues that reporting only total test-time compute can conflate ceiling-raising training with bounded search.Source: §G.4 · pp. 40–41
  • The saturation pattern is empirically supported, but the evidence does not establish that a fixed reachable class causes it; a fixed-weight experiment varying scaffolds remains needed.The paper identifies this as the decisive experiment not delivered by the present data.Source: §G.4 · p. 42

G.5 Bounded vs. Unbounded Metrics

For bounded quality, the paper derives strict diminishing returns from the finite improvement budget. Without boundedness, the theorem no longer applies, although several unformalized mechanisms are still expected to promote plateauing.

G.5 Bounded vs. Unbounded Metrics

  • With quality bounded by 1, positive improvements sum to a finite total, so their terms must shrink toward zero and strict diminishing returns follow.This is the boundedness-dependent result for the Stationarity Dichotomy’s fixed-class regime.Source: §G.5 · p. 42
  • For unbounded metrics, the strict theorem breaks down; the paper only argues that overfitting, gradient depletion, and reward hacking may still promote plateauing.These factors are not formalized in the paper, and reward hacking corrupts measurement rather than limiting capability.Source: §G.5 · p. 42

G.6 Remaining Limitations

The limitations span untested feedback architectures, assumptions and scope boundaries in the theory and evaluations, and an unclear operational boundary between reasoning and scaffold expansion. Future work proposes broader validation and a harness that implements the protocol directly; the formalization index distinguishes machine-checked results from remaining unformalized items.

G.6 Remaining Limitations

  • A separate reviewer architecture remains untested, while the theory uses binary success labels, which the authors say align with software engineering’s conjunctive quality gates.The binary assumption is introduced where AdaBoost’s weight update requires it; the Lean core identities also cover real-valued labels and hypotheses.Source: §G.6 · pp. 42–43
  • The formalization encodes statistical results as hypotheses rather than deriving them, though the stationarity dichotomy and frozen-weight ceiling are fully machine-checked.The margin bound in Theorem 12 and expectation properties in Theorem 2 are hypotheses; Corollary 15 is proved using Mathlib’s probability measure and finite-union-bound results.Source: §G.6 · p. 43
  • Part B is a single-vendor, pytest-native regime study rather than primary empirical evidence, and its task pool omits the two largest Lite repositories.The authors identify importing SWE-bench’s per-repository test commands and restoring those repositories as the first replication step.Source: §G.6 · p. 43
  • Production edge is unmeasured, and the survivor-ratio proxy captures structural convergence rather than functional correctness.The claimed production convergence guarantee is conditional on positive edge under adversarial reweighting; equal ratios can describe qualitatively different sessions.Source: §G.6 · p. 43
  • The theory assumes expected per-round improvement, while its binary scaffold-expansion assumption lacks a principled operational boundary.The authors note that running a new test can make previously unvalidatable patches checkable without changing weights, blurring reasoning and tool acquisition.Source: §G.6 · p. 43

G.7 Future Directions

  • Future work includes broader company and harness-verified validation, a separate-reviewer ablation, agent selection, and a harness that directly implements reweighting and weighted voting.The proposed harness would up-weight failed specs and combine patches by weighted vote rather than sequential composition.Source: §G.7 · p. 44

H Formalization Index

  • The formalization tables map core results to Lean declarations, while statistical content remains explicit as hypotheses.The tables list principal declarations, supporting lemmas, and non-vacuity witnesses.Source: H Formalization Index · p. 44, Table 4 · p. 45, Table 5 · p. 46
  • The multi-round weighted-vote run and the converse direction of AdaBoost’s minimax duality remain unformalized.The proposed run additionally requires excluding a perfect pool member because Corollary 11 assumes positive error.Source: H Formalization Index · p. 46
Loading 2609.34924v1…