PaperlayerView exact revision on arXiv

Source-linked AI summary

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

Myeongsoo Kim, Dingmin Wang, Siwei Cui, Farima Farmahinifarahani, Terry Yue Zhuo, Shweta Garg, Baishakhi Ray, Rajdeep Mukherjee, Varun Kumar

arXiv:2603.24631v2cs.SEcs.AI

TL;DR

Pass@1 reports whether code agents resolve issues but does not show where their trajectories succeed or fail. TRAJEVAL compares search, read, and edit stages with a reference patch across 16,758 trajectories. Many capable-model failures reach the correct functions but still produce incorrect patches, with Coherence Collapse the dominant Edit-Quality pattern.

  • Problem

    Pass@1 gives an outcome-based view of code-agent performance, while the evaluation toolkit provides little process-level evidence about where trajectories succeed or fail.

    Source: §1 · p. 1

  • Method

    TRAJEVAL decomposes agent trajectories into reference-patch-aligned search, read, and edit stages and measures stage-wise precision and recall.

    Source: §1 · p. 1

  • Results

    60–69% of capable-model failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches; Coherence Collapse is the largest Edit-Quality theme on SWE-bench Verified and PolyBench Verified.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Five intermediate patches bit-identical to gold passed the SWE-bench Verified Docker harness after an edit-commit checkpoint, while reference-free consensus yielded a directional +3.0 pp Pass@1 on GPT-5 (p=0.08).

    Source: §1 · p. 2

    TRAJEVAL requires a reference patch and is not applicable at deployment time; its localization metrics do not measure code correctness.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).

1 Introduction

Pass@1 records whether a code agent succeeds but does not reveal where its trajectory succeeds or fails. TRAJEVAL compares search, read, and edit stages with a reference patch, exposing patch-quality failures and coherence collapse among capable models.

1 Introduction

  • TRAJEVAL decomposes trajectories into file-level search, function-level read, and function-level edit, measuring stage-wise precision and recall against a reference patch.The framework is training-free and architecture-agnostic, and is applied to 16,758 trajectories across three architectures and seven models.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 60–69% of capable-model failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches.The pattern also persists for the GPT-5 family and Qwen3-235B on bash-only LiveSWEAgent.Source: §1 · p. 1
  • Coherence Collapse is the dominant Edit-Quality pattern, as agents reach correct code and then overwrite or thrash it.It replicates across SWE-bench Verified and multilingual PolyBench Verified, with a length-independent subtype dissociated from context-window degradation.Source: §1 · pp. 1–2
  • Five intermediate edits bit-identical to gold passed the full SWE-bench Verified test suite when resubmitted to the Docker harness.A reference-free consensus variant measured a directional +3.0 pp Pass@1 on GPT-5 + SWE-Agent (p=0.08, n=500).Source: §1 · p. 2

2 TRAJEVAL: A Trajectory Analysis Lens

TRAJEVAL compares an agent’s trajectory and candidate patch with the reference patch to measure where the agent’s search, reading, and edits align. Its reference-based localization measures complement outcome metrics but do not establish semantic correctness.

2 TRAJEVAL: A Trajectory Analysis Lens

  • TRAJEVAL extracts stage-specific features from an execution trajectory and candidate patch by comparing them with a reference patch.The procedure is lightweight, training-free, and architecture-agnostic.Source: §2.1 · p. 2, §2 · p. 2

2.2 Three-Stage Decomposition

  • Search measures file-level localization, Read measures function-level inspection, and Edit measures function-level modification against the reference patch.The stages distinguish navigation, comprehension, and modification without conflating them.Source: §2.2 · p. 3
  • Search precision and recall compare viewed files with gold-modified files; high recall indicates file discovery, while high precision reflects focused exploration.Source: §2.2 · p. 3
  • Read recall distinguishes trajectories that inspect relevant function bodies from those that open a file but miss the buggy region.Source: §2.2 · p. 3
  • Edit metrics measure localization accuracy, not patch correctness; an agent can achieve perfect edit recall while producing an incorrect patch.Semantic correctness is evaluated separately via Pass@k.Source: §2.2 · p. 3

2.3 Golden Context

  • The Tier-0 golden context averages 1.4 files and 2.1 functions per instance, using a tight reference so low precision reflects exploration beyond the patch’s needs.The authors report that a one-hop call-graph-expanded Tier-1 reference supports this design choice empirically; alternative valid fixes touching disjoint locations still receive low recall.Source: §2.3 · p. 3

2.4 Feature Extraction

  • TRAJEVAL records where an agent operates, not whether its code is correct; debugging behaviors such as test interpretation and verification appear only indirectly.It is designed to complement, not replace, Pass@1.Source: §2.4 · p. 4

3 Experiments

The experiments span seven models, three agent architectures, and two verified benchmarks, with trajectories analyzed through a failure-mode partition, an Edit-Quality taxonomy, and end-to-end intervention tests.

3.1 Experimental Setup

  • The study evaluates seven models from 8B to 480B parameters across SWE-bench Verified and multilingual PolyBench Verified.The benchmarks contain 500 Python issues and 382 multilingual instances, respectively.Source: §3.1 · p. 4
  • Three architectures—SWE-Agent, OpenHands, and bash-only LiveSWEAgent—are run with their own default trajectory budgets and harness configurations.The diversity is intended to help isolate model-intrinsic behavioral patterns from scaffold artifacts.Source: §3.1 · p. 4
  • 16,758 trajectories comprise 9,500 SWE-bench and 7,258 PolyBench runs, after two models were excluded from LiveSWEAgent because of rate-limit and timeout errors.The authors state that the central capable-model finding is unaffected by those exclusions.Source: §3.1 · p. 4

3.2 Evaluation Protocol

  • The protocol classifies failures using cascading recall thresholds, decomposes Edit-Quality failures with overlap and edit-count signals, and tests two interventions end-to-end.The interventions are edit-commit checkpointing on five exact intermediate edits and reference-free parallel-sample consensus on 500 SWE-bench instances.Source: §3.2 · p. 4

4 Results

TRAJEVAL’s stage-level analysis finds that capable-model failures often reach and edit the correct functions yet still produce incorrect patches. A taxonomy of these Edit-Quality failures identifies Coherence Collapse, and the paper tests checkpointing and consensus interventions.

4.1 Failure Mode Analysis

  • 64–69% of failures across five capable models on SWE-Agent are Edit-Quality, and the pattern remains dominant on OpenHands and for most capable models on bash-only LiveSWEAgent.On LiveSWEAgent, Qwen3-Coder-480B instead shifts to Read-dominant failures, while Qwen3-Coder-30B becomes Search-modal.Source: §4.1 · p. 5

4.2 Qualitative Taxonomy of Edit-Quality Failures

  • 39.7% of 914 SWE-Agent Edit-Quality failures on SWE-bench Verified are Coherence Collapse, where agents abandon or break earlier progress; Confused Thrashing accounts for 29.1% and Near-Correct Corrupted for 10.6%.The five themes are assigned using trajectory signals and a cascading-precedence rule; Coherence Collapse and Semantic Error are distinguished operationally by token overlap with the gold edit.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • 5 Near-Correct Corrupted trajectories contained an intermediate patch identical to the gold reference before the agent destroyed it later.Across 97 cases, 30 reconstructed gold-patch hunks at EXACT or NEAR level, and 96 targeted the gold code region.Source: §4.2 · p. 6, Table 3 · p. 6
  • Coherence Collapse rises across trajectory-length quartiles, but Near-Correct Corrupted is length-independent (ρ = 0.001).The aggregate increase is driven almost entirely by Confused Thrashing; longer trajectories add intervening steps before corruption, not a greater probability of corruption.Source: §4.2 · p. 6
  • Semantic Error accounts for 33.0% of Edit-Quality failures, while Intra-File Mislocalization is 17.6%, Scope/Completeness 7.9%, and Execution Gap 1.8%.Independent labeling yields substantial agreement on the headline three-class distinction (κ = 0.80; 90% raw agreement).Source: §4.2 · p. 6
  • 32.3% of PolyBench Verified Edit-Quality failures are Coherence Collapse; JavaScript is 42.1% and Java 36.3%, so the pattern is not specific to Python or SWE-bench.PolyBench Python’s 26.4% rate drives the aggregate drop.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Falsifiable Predictions from the Diagnosis

  • All 5 EXACT intermediate edits pass the full SWE-bench test suite when frozen and re-submitted to the Docker harness, establishing an edit-commit checkpointing existence proof.The diagnosis distinguishes this agent-loop intervention from code-generation interventions suggested for Semantic Error.Source: §4.3 · p. 7
  • +3.0 pp Pass@1 on GPT-5 under consensus-driven feedback is directional but sub-threshold (69.0% vs. 66.0%; McNemar’s p = 0.08, n=500).CONSENSUS3 has 52.4% precision, so roughly half of triggered confirmations target non-gold files and can misdirect the agent.Source: §4.3 · p. 7, Table 5 · p. 7

5 Related Work

Related work motivates process-level evaluation because outcome metrics discard trajectory information, while distinguishing TRAJEVAL’s descriptive, reference-patch-aligned analysis from training-based and prescriptive approaches. The paper positions its contribution in the diagnoses and validated interventions it derives.

5 Related Work

  • Prior reference-patch work uses patches for teacher-trajectory scoring or PRM training, whereas TRAJEVAL is a training-free evaluation procedure focused on diagnoses and validated interventions.The paper names Ma et al. (2026a) and Dihan and Khan (2026) as concurrent work.Source: §5 · p. 8
  • Outcome-only metrics discard trajectory information; TRAJEVAL is an evaluation analogue to process reward models, complementary to PRM-style supervision rather than a replacement.The cited process-supervision works are Wei et al. (2022), Huang et al. (2022), and Schick et al. (2023); the PRM works include Ma et al. (2026a) and Dihan and Khan (2026).Source: §5 · p. 8
  • Unlike action-type trajectory analyses, TRAJEVAL aligns behavior to reference-patch targets; unlike Agentless, its search/read/edit decomposition describes existing agents rather than prescribing an architecture.The paper also distinguishes its per-trajectory diagnostic scope from cross-framework rule-transfer studies.Source: §5 · p. 8

6 Conclusion

TRAJEVAL shows that capable-model failures often persist after agents reach and edit the correct functions, with Coherence Collapse the dominant SWE-Agent pattern. The paper tests checkpoint and consensus interventions and identifies follow-up analyses.

6 Conclusion

  • 60–69% of capable-model failures on SWE-Agent and OpenHands are Edit-Quality failures, and the pattern persists on LiveSWEAgent; Coherence Collapse is the largest SWE-Agent theme at 39.7% on SWE-bench and 32.3% on PolyBench.Near-Correct Corrupted is length-independent and not reducible to context-window degradation.Source: §6 · p. 8
  • An edit-commit checkpoint recovers five intermediate edits bit-identical to gold against the SWE-bench Docker harness, while reference-free consensus yields a directional +3.0 pp Pass@1 on GPT-5 (p=0.08).The passage presents checkpoint recovery as an existence proof; the consensus result is directional.Source: §6 · p. 8
  • The paper identifies taxonomy replication on OpenHands and LiveSWEAgent and trajectory summarization for Confused Thrashing as immediate follow-ups.It also releases the trajectory-extraction pipeline, per-trajectory features, and checkpoint-validation artifacts.Source: §6 · p. 8

Limitations

The paper’s diagnostics depend on reference patches and measure localization rather than semantic correctness. Its consensus result, taxonomy construction, and cross-scaffold replication also have bounded evidence or scope.

Limitations

  • TRAJEVAL requires a reference patch, is not applicable at deployment time, and can assign low recall to valid fixes that modify different locations.Among 350 multiply-solved instances, 100% showed overlapping function-level modifications and 93.7% exact convergence; Tier-1 reference expansion preserved model rank ordering but reduced failure-partition discrimination.Source: Limitations · p. 9
  • Search, read, and edit capture where agents operate, not whether their patches are correct, so TRAJEVAL complements rather than replaces Pass@1.Debugging behaviors such as test interpretation and post-edit verification appear only indirectly through their effects on stage-wise recall.Source: Limitations · p. 9
  • The +3.0 pp consensus measurement covers only GPT-5 with SWE-Agent on SWE-bench Verified and is sub-threshold at α=0.05.At n=500, the study reports 50% power to detect +3 pp and 80% power for +5 pp; larger or pooled measurements would tighten the confidence interval.Source: Limitations · p. 9
  • The Edit-Quality taxonomy uses heuristic thresholds selected through single-annotator calibration, with substantial agreement on the three-class distinction (κ=0.80; 90% raw agreement) but lower agreement on the six-category rule (κ=0.68).Disagreements concentrate on Scope/Completeness; they do not affect the five Near-Correct Corrupted EXACT cases, which were independently validated.Source: Limitations · p. 9
  • The Edit-Quality taxonomy is computed on SWE-Agent trajectories, leaving replication on OpenHands and LiveSWEAgent for future work.Source: Limitations · p. 9

Ethics Statement

The study analyzes automated-agent trajectories from public open-source benchmarks, without human subjects, personal data, or private repositories. It flags non-deployable oracle signals and possible benchmark gaming, and describes safeguards.

Ethics Statement

  • The study uses automated trajectories from SWE-bench Verified and PolyBench Verified, with no human subjects, personal data, or private repositories.The repositories are open-source and used under their original licenses; the trajectories predate the study, and TRAJEVAL runs on CPU in minutes.Source: Ethics Statement · p. 9
  • The paper cautions that Section 4 oracle signals are non-deployable and the consensus measurement covers one model, scaffold, and benchmark cell, so neither guarantees end-user performance.Source: Ethics Statement · p. 9
  • Stage-level diagnostics could be used to game benchmark leaderboards by matching reference-patch trajectories, so the authors release the pipeline as an evaluation tool rather than a training signal and distinguish localization from semantic correctness.Source: Ethics Statement · p. 9

A.1 Appendix Overview

The appendix provides supplementary results, validation, methodological details, and robustness analyses supporting the main findings.

A.1 Appendix Overview

  • The appendix reports the PolyBench cross-benchmark taxonomy and language slices, taxonomy assignment and threshold robustness, and inter-annotator agreement at three levels.Source: §A.1 · p. 11
  • Robustness analyses cover alternative correct solutions through convergence analysis and reference-tier sensitivity.Source: §A.1 · p. 11
  • Supplementary failure-mode breakdowns cover remaining scaffold-benchmark cells and include cross-agent behavior and over-exploration metrics.Source: §A.1 · p. 11
  • Validation materials include Near-Correct Corrupted diff-hunk checks, five-of-five checkpoint cases passing, consensus-as-relevance-signal validation, and a trajectory-length dissociation analysis.Source: §A.1 · p. 11
  • Additional materials provide Edit-Quality worked examples, complete efficiency and tool-use metrics, and trajectory feature-extraction methodology.Source: §A.1 · p. 11

A.2 Cross-Agent Behavioral Patterns: Full Tables

The SWE-bench Verified behavioral and efficiency metrics show shared difficulty locating relevant files, variation across later stages, and substantial over-exploration.

A.2 Cross-Agent Behavioral Patterns: Full Tables

  • Search recall averages 0.72–0.78 across agents, indicating that locating correct files is a shared challenge.Table 6 reports the per-agent and per-model behavioral metrics.Source: §A.2 · p. 11, Table 6 · p. 11
  • Read recall ranges from 0.58–0.71 and Edit recall from 0.45–0.68, while precision is uniformly low across stages and agents.OpenHands has the highest Read recall, while SWE-Agent performs best on Edit recall.Source: §A.2 · p. 11
  • Agents inspect 8–12× more files and 17–25× more functions than necessary; Qwen3-235B has a 26.5% submit success rate despite near-perfect view reliability.These operational metrics are reported for SWE-bench Verified.Source: §A.2 · p. 11, Table 6 · p. 11

A.3 Edit-Quality Themes by Model

The table organizes Edit-Quality failures by model and names five failure themes, with percentages calculated within each model’s Edit-Quality failures.

A.3 Edit-Quality Themes by Model

  • Table 7 reports each model’s Edit-Quality failure themes as percentages of that model’s Edit-Quality failures.The table covers five named themes.Source: Table 7 · p. 11
  • Coh. denotes Coherence Collapse, Sem. denotes Semantic Error, and Loc. denotes Intra-File Mislocalization.Source: Table 7 · p. 11
  • Scope denotes Scope/Completeness, while Gap denotes Execution Gap.Source: Table 7 · p. 11

A.4 Cross-Benchmark Taxonomy: PolyBench Verified

This section examines Coherence Collapse across PolyBench Verified and appendix analyses, covering its cross-language pattern and the Python-concentrated aggregate drop. It also evaluates taxonomy robustness, trajectory partitioning, checkpointing and consensus signals, subtype trajectory lengths, and examples of correct edits later corrupted.

A.4 Cross-Benchmark Taxonomy: PolyBench Verified

  • Coherence Collapse reaches at least 32% in five language-benchmark cells; JavaScript is 42.1% and Java 36.3%, versus 39.7% for SWE-bench Python.These comparisons provide evidence against the finding being a Python- or SWE-bench-only artifact.Source: §A.4 · p. 12, Table 9 · p. 12
  • The 7.4 pp aggregate Coherence Collapse drop on PolyBench is concentrated in Python (26.4%), 13.3 pp below SWE-bench Python, and attributed to benchmark composition rather than language.PolyBench Python issues differ in repository selection, issue type, and test structure; Java and JavaScript show higher Coherence Collapse than the SWE-bench baseline.Source: §A.4 · p. 12, Table 9 · p. 12
  • Execution Gap rises from 1.8% to 8.6%, concentrated in PolyBench Python (12.0%) and TypeScript (14.3%), through code paralysis or fix misattribution.In 10 manually inspected Python cases, 4 showed paralysis and 6 misattribution; the pattern involves reading the correct code but failing to edit it, not a parser artifact.Source: §A.4 · p. 12

A.5 Robustness to Alternative Correct Solutions

  • Across 350 multiply-resolved instances, all successful models edit at least one common function in 100% of cases, and 99.4% have overlapping files.93.7% show exact function-level convergence. Yet 106 successful resolutions (8.6%) have edit recall below 0.8, including partial completions and alternative code paths that pass all tests.Source: §A.5 · pp. 12–13

A.6.3 Tool Success Rates; A.7.3 Tool Success Rates

  • Qwen3-235B has 26.5% submit success on SWE-bench versus 99.9% for GPT-5, while Qwen3-32B’s create, str_replace, and undo_edit success rates degrade further on PolyBench.Qwen3-32B’s PolyBench rates are 58.7% create, 83.5% str_replace, and 43.8% undo_edit; Qwen3-235B submit success improves to 65.3% there but remains below other models.Source: §A.6.3 · p. 13, §A.7.3 · p. 16

A.6.1 Efficiency Metrics; A.7.1 Efficiency Metrics; A.7.2 Token Consumption by Tool

  • GPT-5 averages 38.38 tool calls per PolyBench SWE-Agent task versus 30.71 on SWE-bench, while PolyBench bash output and edit contexts expand substantially.GPT-5-mini reaches 21,460 bash output tokens on PolyBench versus 1,497 on SWE-bench; GPT-5 str_replace output is 1,396 versus 316 tokens.Source: §A.7.1 · p. 13, §A.7.2 · pp. 13, 16

A.8.2 PolyBench Verified; A.9.2 PolyBench Verified

  • OpenHands and LiveSWEAgent also show PolyBench-specific changes: GPT-5 uses 20% more OpenHands tool calls, while LiveSWEAgent success remains 93.3–97.9%.OpenHands GPT-5 fsWrite success drops from 89.0% to 66.7%; LiveSWEAgent’s bash-only operation retains high success across models and benchmarks.Source: §A.8.2 · pp. 16, 19, §A.9.2 · pp. 19, 21

A.10 Algorithm: Trajectory Feature Extraction; A.11 Trajectory Feature Extraction Methodology; A.12 Threshold Sensitivity for Failure-Mode Buckets

  • The feature-extraction pipeline computes six trajectory metrics and defines golden context as the minimal code elements required to generate the correct patch: golden files, functions containing modified lines, and modified line ranges.The failure-mode partition uses a cascading 0.5 threshold on stage-wise recall; the same SWE-Agent + SWE-bench failures are re-bucketed at τ ∈ {0.3, 0.5, 0.7} to test whether qualitative findings depend on that choice.Source: §A.11 · p. 21, §A.11.1 · p. 21, §A.12 · p. 22

A.12 Threshold Sensitivity for Failure-Mode Buckets

  • Across threshold choices, capable models remained Edit-Quality-dominated at 53–72%, while the τ=0.7 shift mostly moved labels from Edit-Quality to Search.Qwen3-8B and Qwen3-32B remained upstream-failure models, with Search + Read at least 50% at τ=0.7; stricter thresholds changed labels rather than underlying behavior.Source: §A.12 · p. 22

A.13 Reference-Tier Sensitivity: Tier-0 vs. Tier-1 Golden Context

  • Mean Search recall fell from 0.756 under Tier-0 to 0.292 under Tier-1, shifting 4,379 trajectories into Search failure and erasing downstream-bucket discrimination.Despite this absolute sensitivity, model rankings by aggregate recall were identical (Spearman ρ = 1.00); Tier-0 measures files actually modified, while Tier-1 includes one-hop dependencies.Source: §A.13 · p. 23

A.14 Failure-Mode Distribution: Remaining (Scaffold, Benchmark) Cells

  • 60–70% of capable-model SWE-bench failures were Edit-Quality failures across scaffolds, while PolyBench shifted the modal failure upstream.On PolyBench, Search dominated for SWE-Agent (47.0% weighted average) and Read for LiveSWEAgent (52.8%); smaller Qwen variants had higher Search-failure rates across cells. The tables report the underlying scaffold-benchmark distributions.Source: §A.14 · p. 24, Table 31 · p. 24, Table 32 · p. 24

A.16 Assumptions and Limitations

  • The intervention findings are limited in scope: checkpointing is validated on five EXACT cases, while consensus was tested only on GPT-5 with SWE-Agent on SWE-bench Verified.The consensus result was +3.0 percentage points (p = 0.08, n=500), below α = 0.05; deployment-grade checkpointing and broader replication remain open, and generalization beyond SWE-bench and PolyBench requires validation.Source: §A.16 · p. 25

A.18 Edit-Quality Taxonomy: Assignment Rule and Threshold Robustness

  • Under threshold perturbations, Coherence Collapse remained the largest or second-largest theme at 36–56%, and model-level rate ordering was preserved.The taxonomy assigns each Edit-Quality failure one label by cascading deletion overlap, token overlap, and gold-file edit count; Scope/Completeness stayed below 10% across configurations.Source: §A.18 · pp. 25–26

A.19 Inter-Annotator Agreement on the Cascading Rule

  • At the headline 3-class level, annotators agreed at κ = 0.80 (90% raw), and all remaining disagreements had the rule label fewer Coherence Collapse cases than the human annotator.Agreement was κ = 0.71 (83% raw) for the 5-class refinement and κ = 0.68 (77% raw) for the full six-category rule.Source: §A.19 · pp. 26–27

A.20 Near-Correct Corrupted: Diff-Hunk Validation

  • 30/97 Near-Correct Corrupted cases reconstructed gold-patch hunks at EXACT or NEAR strength, and 96/97 targeted the gold patch’s code region.The median gap between the near-correct and overwriting or corrupting edits was 6 trajectory steps; GPT-5 had the largest strong-match share per Edit-Quality failure among the five EXACT cases, at 57%.Source: §A.20 · p. 27

A.21 Edit-Commit Checkpoint Validation

  • 5/5 EXACT intermediate edits passed the full SWE-bench Verified test suite, showing an edit-commit checkpoint could recover solutions later corrupted or overwritten.This is a strict existence proof; the realistic recovery ceiling among Near-Correct Corrupted cases remains open, with at most 30 EXACT-or-NEAR cases.Source: §A.21 · pp. 27–28
  • Five EXACT intermediate patches passed tests by construction, establishing checkpointing as deployable; validation of 25 NEAR cases remains confounded by diff-reconstruction errors.The realistic ceiling within Near-Correct Corrupted is at most 30 EXACT + NEAR cases; end-to-end re-execution with a test-triggered freeze remains future work.Source: §A.21 · p. 28

A.22 Consensus-as-Relevance-Signal Validation

  • The consensus signal recovered 97.3% of oracle-triggering cases at N=3, implying an analytical ceiling of +4.5 pp Pass@1 lift only under perfect-precision triggering.Precision was 52.4% at N=3; at N=5, precision rose to 75.6% while recovery fell to 82.4%. The reference-free leave-one-out signal requires multiple samples, and its validation used the same 500 instances as the trajectory collection.Source: §A.22 · pp. 28–29
  • +15 net instances were gained by the end-to-end consensus intervention, with 39 gained and 24 lost across 500 SWE-bench Verified instances.At N=3, 52.4% precision helps explain the losses: roughly half of confirmatory triggers occur on files outside the gold set, and agents sometimes edit the wrong location.Source: §A.22 · p. 29

A.23 Trajectory Length and Coherence Collapse

  • Coherence Collapse correlated with trajectory length (ρ = 0.32, p = 3.4 × 10−23), but the increase came from Confused Thrashing, not Near-Correct Corrupted.Confused Thrashing rose from 10.5% in Q1 to 52.9% in Q4, while Near-Correct Corrupted stayed near 10–11% across quartiles; its rate had no correlation with length (ρ = 0.001, p = 0.98). Longer trajectories were associated with more steps between near-correct edits and corruption, not a greater probability of corruption.Source: §A.23 · p. 29, Figure 2 · p. 30

A.24 Edit-Quality Worked Examples

  • Three worked trajectories illustrate the dominant Coherence Collapse subtypes among the 914 Edit-Quality failures.The examples cover Confused Thrashing and additive and destructive forms of Near-Correct Corrupted.Source: §A.24 · p. 29

A.24.1 Worked Example 1: Confused Thrashing

  • In a Confused Thrashing example, GPT-5 reached the correct file and code region but repeatedly overwrote edits, then abandoned the approach for an unrelated file.After stale-context replacement attempts, the final patch edited django/db/migrations/serializer.py rather than preserving a fix to the gold file.Source: §A.24.1 · p. 30

A.24.2 Worked Example 2: Near-Correct Corrupted (Additive)

  • In an additive corruption example, a correct import edit was followed by an unnecessary .figure dereference that made the final patch fail tests.The turn-48 edit matched the gold patch’s first hunk; the turn-70 refinement introduced the failure.Source: §A.24.2 · p. 30

A.24.3 Worked Example 3: Near-Correct Corrupted (Destructive Overwrite)

  • In a destructive overwrite example, Qwen3-32B produced a gold-identical, test-passing edit, then deleted db_collation in a later edit and broke CharField initialization.The final patch contained only a reproduce_error.py script, with no productive source-file diff.Source: §A.24.3 · p. 31

Related papers

Loading 2603.24631v2…