PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories

Jiajun Wu, Leixin Sun, Zihan Tan, Yitao Liu, Shuo Li, Jiaru Qian, Shanghaoran Quan, Chuangxin Zhao, Yangxu Liao, Yang Liu, Bin Chong, Guancheng Wan

arXiv:2609.29465v1cs.AIcs.SE

TL;DR

Issue-driven repository benchmarks test whether agents repair known defects, but do not assess broader repository governance. SWE-Prometheus asks agents to diagnose and retrofit fixed repositories, then evaluates paired evidence, behavior preservation, and governance change. Results show varied improvement and breakage, while controls reveal that templates, gate strength, and score reliability affect interpretation.

  • Problem

    Existing repository benchmarks typically begin with a human-identified issue and evaluate a patch against a functional signal, rather than repository-wide engineering governance.

    Source: §1 · p. 1

  • Method

    SWE-Prometheus evaluates open-ended repository retrofits across six governance dimensions using paired base-to-treated evidence, clean-environment probes, behavior gates, and independent teacher ratings.

    Source: §1 · p. 3, §2.5 · p. 5

  • Results

    Across ten models on 22 public repositories, mean NGI ranges from 0.0568 to 0.5760 and observed behavior-breakage rates range from 0% to 23%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Repository-governance evaluation should report improvement, behavior preservation, evidence quality, and coverage together.

    Source: §8 · p. 16

    Each model-instance pair has only one agent rollout, and external validity is limited by the 60-repository sample and its language and ecosystem distribution.

    Source: §6.4 · p. 14

Abstract

from arXiv · show

Large language model based coding agents have made substantial progress on repository-level software engineering tasks. Existing repository benchmarks, however, usually start from a human-identified issue and evaluate whether a patch satisfies a functional signal. We present SWE-Prometheus, a benchmark for the broader task of improving repository engineering governance. Each task provides a fixed snapshot and an open-ended objective, requiring the agent to identify risks, prioritize interventions, and verify the resulting changes. SWE-Prometheus evaluates six governance dimensions through paired evidence, clean-environment probes, behavior gates, and two independent teacher ratings of the same evidence. The benchmark contains 60 repositories; ten models are evaluated on a shared 22-repository public subset, where mean Normalized Governance Improvement ranges from 0.0568 to 0.5760 and observed behavior-breakage rates range from 0% to 23%. On a frozen ten-repository batch, a repository-blind template obtains mean NGI 0.272, but its gains concentrate in Tests & CI, Quality Gates, and Documentation; it improves Reproducible Environment and Dependency & Security on none of the repositories. This baseline makes the distinction between adding governance artifacts and producing execution-backed improvements measurable. The no-op condition has median NGI zero and standard deviation 0.073; two teachers agree exactly on 57 of 60 dimension scores for the same no-op evidence. For the two highest conditional-mean systems, common-valid NGI is similar, while full-pool comparisons that include behavior failures favor Kimi-K3. These results show why repository-governance evaluation should report improvement, behavior preservation, evidence quality, and coverage together.

1. Introduction

SWE-Prometheus reframes repository evaluation from repairing a human-identified defect to diagnosing and improving engineering governance. It measures whether changes yield paired, execution-backed improvements while preserving behavior.

1. Introduction

  • Issue-level benchmarks measure patches against known defects, leaving repository-wide readiness, maintainability, reproducibility, and security outside the target.These governance properties are distributed across repository components and cannot be established from a diff alone.Source: §1 · p. 1
  • SWE-Prometheus gives agents an open-ended retrofit objective: identify consequential gaps, prioritize interventions, and improve a fixed repository snapshot without a prescribed defect or patch.The evaluator reconstructs base and treated states and verifies changes in a clean environment.Source: §1 · pp. 1–2
  • Across ten models on 22 public repositories, mean NGI ranges from 0.0568 to 0.5760, while observed behavior-breakage rates range from 0% to 23%.A repository-blind template reaches mean NGI 0.272 but improves Reproducible Environment and Dependency & Security on none of ten repositories.Source: §1 · p. 2
  • Six governance dimensions are assessed with paired evidence, clean-environment probes, behavior gates, and independent teacher ratings of the same evidence.The dimensions include Tests & CI, Code Quality Gates, Documentation & Collaboration, Structure & Maintainability, Reproducible Environment, and Dependency & Security Health.Source: §1 · pp. 2–3
  • Diagnostic controls distinguish governance improvement from template effects, behavior risk, and verifier limitations.The reported no-op condition has median NGI zero and standard deviation 0.073; teachers agree exactly on 57 of 60 scores for the same no-op evidence.Source: §1 · pp. 2–3

2. Related Work

Prior work supplies repository benchmarks, agent interfaces, executable evaluation methods, software-quality measures, and benchmark-calibration practices. SWE-Prometheus combines these concerns in an open-ended evaluation of repository governance rather than known-issue repair.

2.1. Code and Software Engineering Benchmarks

  • Repository benchmarks broadened issue resolution across languages, modalities, and domains, but retain the endpoint of resolving a known defect.SWE-Prometheus instead evaluates diagnosing and improving a repository’s engineering state.Source: §2.1 · p. 3

2.2. Software Engineering Agents

  • Agent studies show that interaction interfaces and tool-use pipelines matter, while most assume the issue, repair objective, or evaluation signal is already specified.SWE-Prometheus asks agents to discover the target and justify an intervention with evidence.Source: §2.2 · pp. 3–4

2.3. Evaluation Beyond Patch Correctness

  • Executable evaluation, mutation testing, and behavioral testing motivate treating preservation evidence and gate strength as first-class outcomes, beyond whether a new check passes.The benchmark also differs from diagnosis work that explains a supplied failure: its agent must proactively identify governance gaps.Source: §2.3 · p. 4

2.4. Software Quality and Engineering Governance

  • Quality standards, complexity and debt measures, and supply-chain guidance offer measurement primitives, but typically audit properties or practices in isolation.SWE-Prometheus uses such tools as probes in paired evaluation rather than as substitutes for repository-wide improvement.Source: §2.4 · p. 4

2.5. Reliable Benchmark Construction and Judge Calibration

  • Benchmark construction and judge research motivate explicit provenance, reproducible execution, retained failed runs, mutation-tested behavior gates, and calibrated judgments.The remaining need is an integrated evaluation in which agents discover and improve repository targets under realistic constraints.Source: §2.5 · pp. 4–5, Figure 2 · p. 5

3. Dataset Overview

The task asks agents to improve engineering readiness across applicable governance dimensions without a supplied issue or reference patch. Evaluation pairs base and treated repository evidence and tracks both governance change and behavior preservation.

3. Dataset Overview

  • Agents inspect a fixed repository snapshot, improve readiness across applicable dimensions, preserve behavior and public interfaces, and verify important claims with executable checks.The task supplies no issue list, base score, hidden test, or reference patch, allowing multiple valid strategies.Source: §3.1 · p. 5
  • Six dimensions are scored from 1 to 5, and configuration presence alone cannot establish improvement.The dimensions cover tests and CI, quality gates, documentation and collaboration, structure and maintainability, reproducible environments, and dependency and security.Source: §3.1 · p. 5
  • The evaluator runs the same installation, test, quality, complexity, dependency, and security probes on base and treated snapshots, retaining logs and environment details.This paired evidence record is scored twice by independent teacher models using the same rubric.Source: §3.1 · p. 6
  • Characterization tests validated on the base are rerun after treatment; failures are labeled behavior_broken, excluded from valid NGI aggregation, and retained for behavior-risk analysis.Mutation testing labels gates detected, blind, vacuous, or none.Source: §3.1 · p. 6
  • A base score of 5 receives no improvement reward but is still checked for regression.The report also tracks treated scores, valid and broken rates, no-regression rate, strict success, verified claim rate, and resource usage.Source: §3.1 · p. 6

4. Dataset Creation

Dataset creation selects reproducible repositories with meaningful behavior and applicable governance gaps, then packages paired evidence and task instructions. A controlled execution and scoring protocol preserves failure evidence and calibrates independent judgments.

4.1. Repository selection and screening

  • Repositories are retained only when their base commit is reconstructable, behavior is executable, clean setup needs no private credentials, at least two dimensions apply, and the task does not prescribe a patch.Candidate discovery signals support high-recall collection; final scores rely on executable evidence, with unavailable or inapplicable evidence recorded explicitly.Source: §4.1 · p. 6

4.2. Evidence collection and task packaging

  • Each repository record includes a fixed identifier and base commit, governance brief, applicable dimensions, baseline evidence, characterization commands, and environment metadata.The brief avoids naming a defect or prescribing a patch, preserving multiple retrofit strategies while keeping tasks reproducible.Source: §4.2 · p. 6

4.3. Execution protocol

  • Rollouts start from fresh copies of pinned commits, and patches are applied to a second clean copy for installation, behavior, governance, and artifact checks.The harness records tool calls, changed files, patch size, time, exposed token usage, termination reason, logs, tool versions, and timeouts.Source: §4.3 · pp. 6–7

4.4. Behavior gates and failure taxonomy

  • Characterization-test failures after treatment are labeled behavior_broken and excluded from valid NGI aggregation, but retained in behavior-risk analysis.Mutation testing distinguishes detected, blind, vacuous, or absent gates; weaker gates should not be treated as strong safety evidence.Source: §4.4 · p. 7

4.5. Scoring and judge calibration

  • Executable probes score tests, CI, environment, and dependencies, while documentation and maintainability use dimension-specific rubrics with positive and negative evidence, applicability, and confidence.Independent judgments are preferred; calibration reports agreement and other checks while preserving initial labels before adjudication.Source: §4.5 · p. 7

5. Results

The study evaluates ten models on a shared 22-repository public subset and uses auxiliary controls to separate model outcomes from template effects, behavior risk, and measurement limits. Results show wide variation across models and repositories, while reliability, evidence coverage, and verifier strength qualify simple score comparisons.

5.4. Reliability-aware comparison of the two leading systems

  • On 17 repositories valid for both systems, Kimi-K3 averages 0.547 NGI versus 0.576 for GLM-5.3-Flash; across full-pool policies, Kimi-K3 ranks first.The common-valid difference is smaller than the no-op standard deviation of 0.073; coverage-weighted NGI is 0.550 for Kimi-K3 and 0.445 for GLM-5.3-Flash.Source: §5.4 · pp. 8–9, Table 2 · p. 9

5.3. Governance improvement varies widely across agents

  • Mean NGI ranges from 0.0568 to 0.5760, while observed behavior breakage ranges from 0% to 22.7% across the 22-repository public subset.Kimi-K3 and GLM-5.3-Flash share the highest valid-run mean NGI but break behavior on one and five instances, respectively.Source: §5.3 · p. 8, Figure 3 · p. 9

5.5. Instance-level dispersion is large relative to the model spread

  • Model means conceal wide repository-level dispersion, and single-rollout results jointly reflect repository heterogeneity, rollout variability, and scorer noise.Claude-Opus-5's auxiliary-batch median of 0.722 exceeds its mean of 0.457, indicating gains concentrated in a subset of instances.Source: §5.5 · pp. 9–10

5.7. Non-agent controls and no-op measurement variation

  • The no-op condition has median NGI 0.000 and standard deviation 0.073; two teachers agree exactly on 57 of 60 no-op repository-dimension scores.Mechanical and rule-based templates score mean NGI 0.272 and 0.254, yet improve Reproducible Environment and Dependency & Security on none of the repositories; Efficient-WAM illustrates how an artifact can earn credit without meaningful coverage or reproducible installation.Source: §5.7 · p. 10

5.8. Matched comparison against the template baselines

  • Only Kimi-K3 and GLM-5.3-Flash beat the stronger template on a clear majority of the ten repositories, with 9–1 and 8–1 wins-losses, respectively.Claude-Opus-5 and Qwen3.8-Max are near even, while DeepSeek-V4-Flash and MiniMax-M3 lose on six and seven repositories.Source: §5.8 · p. 10, Figure 4 · p. 11

5.9. Verifier quality control

  • Measured breakage falls from 13% under detected gates to 8% under blind gates and 0% under vacuous gates, while median NGI shifts only 0.06.Three review passes of the same sample of patches that passed behavior checks judged the changes useful for maintenance and identified no new functional problems; gate-off median changes were at most 0.021, and model ordering remained identical.Source: §5.9 · p. 11

5.10. Sensitivity of the current design

  • At 80% power and α = 0.05, the public subset resolves a true mean difference of about 0.19, versus about 0.27 for the ten-repository auxiliary batch.Adjacent models separated by less than this resolution limit are not distinguished by the instrument; resolving 0.10 would require about 110 paired observations.Source: §5.10 · pp. 11–12, Figure 5 · p. 12

5.11. Per-dimension decomposition

  • Tests & CI improves on most valid runs for every model except MiniMax-M3, while Structure & Maintainability is weakest or jointly weakest for eight of ten models.Both templates improve Tests & CI, Code Quality Gates, and Documentation & Collaboration on nearly every instance, but improve Reproducible Environment and Dependency & Security on none; visible artifacts can diverge from functional evidence.Source: §5.11 · pp. 12–13, Figure 6 · p. 13, §5.12 · p. 13

6. Analysis

SWE-Prometheus distinguishes repository-wide engineering effectiveness from artifact presence and issue-level repair, while treating governance improvement, behavior preservation, and evidence quality as separate concerns. The analysis highlights repository-specific diagnosis as a discriminating capability, but notes that one-shot sampling, scaffold differences, and the repository sample constrain ranking claims.

6.1. What the benchmark makes visible

  • The evidence loop tests whether governance artifacts work meaningfully and preserve behavior; a mechanical template raises D1–D3 but not D5 or D6, and rarely changes D4.The template calibrates which rubric dimensions are accessible through adding conventional files alone.Source: §6.1 · p. 14
  • Per-dimension intervention rates reveal asymmetric progress: tests and environment files are easier to add than dependency security, maintainability, and collaboration practices.Reporting dimensions separately helps prevent template-shaped gains from being mistaken for repository-specific diagnosis.Source: §6.1 · p. 14

6.2. Improvement, safety, and absolute quality are separate axes

  • Kimi-K3’s stronger full-pool verified outcome despite similar common-valid means shows that improvement and behavior-breakage risk are separate axes.The analysis also cautions that higher NGI does not automatically imply lower risk, and zero observed breaks can reflect few changes.Source: §6.2 · p. 14
  • The gate marks scores that cannot be trusted rather than correcting them; gate strength determines how much confidence a zero-breakage observation warrants.Removing the gate changes some model medians in both directions, but does not change model ordering.Source: §6.2 · p. 14

6.3. Implications for agent design

  • Only two of ten systems beat the stronger template on a clear majority of matched repositories, making repository-specific diagnosis a discriminating benchmark capability.Aggregate gains alone do not show that an agent chose a better intervention than a repository-independent template.Source: §6.3 · p. 14
  • Agents should map repositories, prioritize gaps requiring execution, avoid unnecessary source changes, and verify both positive claims and side effects.The authors propose future baselines with explicit diagnosis, planning, patching, and audit phases.Source: §6.3 · p. 14

6.4. Validity threats and mitigations

  • One rollout per model–instance pair and the 60-repository sample constrain validity; Claude-Opus-5’s different scaffold also means cross-scaffold ordering is not causal.The current sensitivity analysis supports relative-improvement and reliability conclusions more strongly than precise conditional-mean rankings.Source: §6.4 · pp. 14–15

7. Limitations and Future Work

The authors identify limits in behavioral gate strength, rubric susceptibility to template gains, ranking resolution, scaffold comparability, and the public repository sample. Planned work includes repeated rollouts, stronger calibration, and a more isolated holdout.

7. Limitations and Future Work

  • Only 24 of 60 instances have behavior gates with mutation-demonstrated discriminative power, so other instances provide weaker behavioral evidence.The authors also note that subjective dimensions may share judge errors and call for broader independent expert calibration.Source: §7 · p. 15
  • Three dimensions are reachable by a repository-blind template, exposing rubric vulnerability unless credit requires demonstrated scope and execution.The authors recommend tightening Tests & CI, Code Quality Gates, and Documentation & Collaboration.Source: §7 · p. 15
  • At the observed paired dispersion, the public subset resolves differences of about 0.19 NGI, so adjacent ranking positions are not separated by the instrument.The single-rollout design bounds ranking claims.Source: §7 · p. 15
  • Different scaffolds expose different tools and prompts, while unavailable monetary-cost data limits efficiency conclusions to tokens, turns, wall-clock time, and tool calls.Source: §7 · p. 15
  • The 22-repository public subset is not randomly sampled, and public repositories may overlap with training data or later repository versions.The authors call for repository-family separation, near-duplicate scanning, contamination audits, and an access-controlled private evaluator.Source: §7 · p. 15
  • D6 measures available dependency and security evidence, not comprehensive vulnerability absence; public releases must also exclude private evaluation artifacts.Artifact manifests should make missing traces or evidence explicit rather than representing them with empty directories.Source: §7 · p. 15
  • Planned work includes repeated rollouts, expert baselines, stronger characterization tests, same-scaffold reruns, judge calibration, and an isolated private holdout.Template and deterministic baselines need expansion across repository families before they can be treated as general lower bounds.Source: §7 · p. 15

8. Conclusion

SWE-Prometheus broadens repository-agent evaluation beyond issue resolution to assess diagnosis, improvement, and behavior-preserving stewardship, with results exposing differences across systems and governance dimensions.

8. Conclusion

  • Ten agent systems evaluated on 22 shared repositories show variation in governance improvement, behavior-regression risk, and dimension-specific preferences.The release contains 60 real repositories; executable evidence can reveal failures or behavior damage that a diff alone may not show.Source: §8 · p. 15
  • The repository-blind template reaches mean NGI 0.272, but gains concentrate in D1–D3 and it makes no gains in D5 or D6.The no-op median is zero with standard deviation 0.073; teachers agree exactly on 57 of 60 dimension scores, and only two of ten systems clearly beat the stronger template on most repositories.Source: §8 · p. 16
  • SWE-Prometheus separates useful governance practices, their operation in the target project, and preservation of existing behavior to distinguish artifact completion from execution-backed improvements.It complements issue-level repair benchmarks with repository diagnosis, prioritization, intervention, and verification.Source: §8 · p. 16

A.1. Artifacts and accounting

  • Instance records pair repository identity and baseline information with patches, base and treated evidence, verification, budgets, and scores; missing artifacts require an explicit reason.The release checklist also includes environment locks, deterministic scorer configuration, logs, evaluator and judge versions, table-reconstruction scripts, and a changelog.Source: §A.1 · p. 21, §A.3.1 · p. 21, §A.3.3 · p. 22

A.2. Rubric anchors

  • Scores of 1–2 denote absent or non-functional governance, 3 partial coverage with material gaps, 4 an executable and reviewable standard, and 5 broad coverage with maintainability evidence.Across D1–D6, a file’s presence without successful execution is insufficient for a score of 4 or 5.Source: §A.2 · p. 21, §A.3.5 · p. 22

A.3.2. Detailed gate-strength accounting

  • The 60-instance release includes 24 mutation-demonstrated gates, 27 blind gates, 3 vacuous gates, and 6 instances without a usable behavior gate.Results distinguish the full set, mutation-demonstrated subset, and no-gate subset.Source: §A.3.2 · p. 22, Table 5 · p. 22

A.3.4. Auxiliary experiment worksheet

  • The frozen ten-repository auxiliary batch omits repeated-seed, same-scaffold, and expert conditions, so those conditions are not represented in its measurements.Completed experiments record their sampling frame, model and scaffold, paired comparison unit, and machine-readable result source.Source: §A.3.4 · p. 22

A. Dataset Construction Details

  • The appendix separates benchmark construction, experiment lifecycle, scoring, analysis, and reproducibility materials, while making fixed, hidden, and quality-control elements explicit.Its expanded rubric prompts judge meaningful tests and CI, enforced quality commands, actionable contributor guidance, maintainable boundaries, reproducible environments, and inspectable dependency-security evidence.Source: §A.4 · p. 22, §A.3.5 · p. 22

A.4.1. Dataset construction overview

The dataset is built from repositories pinned to fixed commits and admitted only after behavior, environment, and governance headroom are checked.

A.4.1. Dataset construction overview

  • The release contains 60 instances: 22 public instances with shared model coverage and 38 private instances reserved for internal evaluation.The public/private split is assigned after task validation and does not alter scoring.Source: §A.4.1 · pp. 22–23

A.4.2. Construction principles

  • Each task starts from a reconstructable snapshot, permits multiple valid interventions, connects improvements to probes or structured judgments, and tests behavior preservation separately.The task, agent patch, and evaluator evidence are distinct; any construction patch is not exposed to the agent as an oracle.Source: §A.4.2 · pp. 23–24

A.4.3. Candidate discovery funnel

  • Candidate discovery combines broad retrieval, mechanical screening, manual validity inspection, and evaluator dry runs to reject broken checkouts and unmeaningful governance gaps.The final dry run rejects irreproducible environments, private-service requirements, absent dimension headroom, or behavior that cannot be characterized.Source: §A.4.3 · p. 24

A.4.4. Baseline assessment and headroom

  • Baseline assessment records probe outputs, normalized scores, applicable dimensions, and score rationales; high headroom alone cannot qualify a non-functional environment.The baseline pass distinguishes project-declared dependencies from tools preinstalled in the evaluator image, especially for D5 and D6.Source: §A.4.4 · p. 24

A.4.5. Behavior-gate construction

  • Behavior gates use repository tests where possible or validated characterization tests, and mutation checks label a gate detected only when it catches a behavior-changing mutation.The gate checks destructive interventions; it raises confidence in an outcome but does not directly increase governance scores.Source: §A.4.5 · p. 24

A.4.6. Split construction and leakage control

  • The public split supports shared comparison and trace inspection, while private tasks and verification assets are withheld from public outputs.The passage identifies repository-family separation, near-duplicate scanning, and temporal contamination checks as additions for a broader public split.Source: §A.4.6 · p. 24

A.4.7. Instance packaging

  • Each packaged instance has task metadata, repository identity, a base commit, applicability mask, base evidence, verification asset, and evaluator configuration.The task does not disclose a target score, preferred file, or reference implementation; execution-affecting omissions are validity concerns.Source: §A.4.7 · pp. 24–25

A.4.8. Quality-control passes

  • Three independent quality-control passes check package integrity, evaluator integrity, and interpretation of scores against evidence and logs.The passes are separated because a package can execute but be conceptually invalid, or be conceptually sound but irreproducible.Source: §A.4.8 · p. 25

B. Detailed Main Experiment

The experiment gives agents an open-ended repository task, then evaluates their patches against paired evidence and behavior verification in a clean snapshot.

B.1. Task tuple

  • An instance combines a repository, pinned commit, task metadata and applicability mask, agent instruction, executable environment, and verification package.The agent receives the repository, commit, metadata, instruction, and environment, but not protected verification assets.Source: §B.1 · p. 25

B.2. Evidence record

  • Each probe record captures its command and environment, timing, exit and timeout status, output digest, normalized result, applicability, and interpretation.The normalized record must explain the score even when the raw log is stored separately.Source: §B.2 · p. 25, Table 9 · p. 26

B.3. Applicability and missingness

  • The scorer records applicability separately from outcome, so an inapplicable dimension or unavailable evidence is not converted into an unexplained zero.Missingness distinguishes not applicable, environment failure, timeout, and malformed artifact.Source: §B.3 · p. 26

B.4. Normalization and aggregation

  • Normalized improvement is computed over defined, applicable dimensions, with dimension counts and raw and treated score sums retained alongside the aggregate.This avoids treating identical NGI values as equivalent when they use different numbers of dimensions.Source: §B.4 · p. 26

B.5.1. Agent-facing instruction template

  • The task asks agents to improve applicable governance dimensions, preserve behavior and interfaces, prioritize using repository evidence, and verify important claims.The instruction is open-ended: agents discover project conventions rather than receive a checklist of missing files.Source: §B.5.1 · p. 26

B.5.2. Run lifecycle

  • The lifecycle reconstructs and verifies the base, launches the model, archives its patch, applies it to an independent clean snapshot, runs probes and behavior checks, then scores the evidence.The agent-side container is discarded so mutable working-tree caches or generated files cannot influence evaluation.Source: §B.5.2 · pp. 26–27

B.5.3. Canonical-run policy

  • Retries are limited to infrastructure failures; valid but incorrect patches are not retried to improve their result.When a retry occurs, the original record remains archived and the canonical run is identified with a reason.Source: §B.5.3 · p. 27

B.5.4. Budget accounting

  • Run budgets record model and scaffold, time limits and elapsed time, available token counts, turns, tool calls, termination reason, and provider cost when available.Incomplete provider accounting is marked unavailable rather than replaced with estimated prices.Source: §B.5.4 · p. 27, Table 10 · p. 27

C. Evaluation and Scoring

The scoring protocol combines evidence-based dimension ratings, separate regression gates, and paired analyses while retaining uncertainty, coverage, and missing-run information.

C.1. Dimension-specific score anchors

  • A score of 4 requires executable or inspectable evidence appropriate to the repository; configuration alone is insufficient.The dimension-specific rules require meaningful tests, relevant linting, actionable prose, improved boundaries, consistent environment metadata, and dependency reports that distinguish evaluator tools.Source: §C.1 · p. 27, §C.2 · p. 27

C.3. No-regression rules

  • A confirmed behavior break cannot be offset by governance improvement in the primary valid-run aggregate.The protocol flags failed characterization behavior, unexpected public-interface changes, or loss of applicable probes.Source: §C.3 · p. 27, Table 11 · p. 28

C.4. Judge calibration worksheet

  • Judge calibration uses a set spanning low, medium, and high baseline quality and records scores, cited evidence, uncertainty, and review judgments before adjudication.Agreement is computed before adjudication, whose changes remain separately recorded.Source: §C.4 · p. 28

C.5.1. Fixed 10-repository batch

  • The auxiliary study uses a frozen ten-repository batch with no-op, agent, mechanical-retrofit, and rule-based conditions; repeated-seed, same-scaffold, and expert conditions were not run.Behavior-gate ablation instead uses all 60 instances because the public subset has no cases without a gate.Source: §C.5.1 · p. 28, §C.5.2 · p. 28, Table 12 · p. 28

C.5.3. Exploratory questions

  • Exploratory paired analyses compare agent rollouts with non-agent baselines and examine filtering, gate strength, template-responsive dimensions, and intervention scope.These descriptive questions use paired instance records where available and are not preregistered hypotheses.Source: §C.5.3 · p. 28, §C.6.1 · p. 29, Table 13 · p. 29

C.6.2. Mean–median reporting rationale

  • Mean NGI is the compact main-table measure, but behavior-broken runs are excluded, so leading systems also receive common-valid and coverage-weighted full-pool comparisons.The 22-instance public comparison has one rollout per model-instance pair, making its mean descriptive rather than a stable population-level capability estimate.Source: §C.6.2 · p. 29

C.6.3. Uncertainty reporting

  • For ten repositories, the analysis emphasizes exact paired differences and repository-unit bootstrap intervals rather than asymptotic significance tests.The batch is explicitly exploratory; repeated seeds add within-model uncertainty when available.Source: §C.6.3 · p. 29

C.6.4. Per-dimension reporting

  • Per-dimension tables report base and treated means, changes, improvement/unchanged/regression fractions, and applicability counts; excluded runs retain codes and pre- and post-exclusion denominators.The appendix provides underlying numbers for the results presented as figures in the body.Source: §C.6.4 · p. 29, §C.6.5 · p. 30, D. Full Result Tables · p. 30

D.1. Public shared-subset results

The public results aggregate NGI over valid runs and report breakage separately; auxiliary analyses compare templates, gate policies, and other governance signals. These comparisons emphasize that conditional improvement, behavior preservation, and measurement scope must be considered together.

D.1. Public shared-subset results

  • Public-subset NGI is aggregated only over valid runs; valid and broken counts are out of 22, and breakage is broken divided by total runs.Claude used Claude Code; the other models used pi.Source: Table 15 · p. 30

D.5. Baseline patch contents

  • The mechanical template writes seven fixed artifacts, while the rule-based version adds only absent artifacts and replaces the vacuous smoke test with an import check.Neither condition uses a language model or modifies existing source files; both had zero behavior-broken runs among ten repositories, an observed batch result rather than a guarantee.Source: §D.5 · pp. 30–31
  • Kimi-K3 ranks first under all three failure-aware policies, while conditional NGI ties it with GLM-5.3-Flash at reported precision.On common-valid repositories, GLM-5.3-Flash has 10 wins, Kimi-K3 has 6, and there is 1 tie.Source: Table 18 · p. 31, Table 17 · p. 31
  • Template baselines improved configuration- and content-evidenced dimensions, but not dimensions substantially informed by container execution and dependency analysis.The per-dimension improvement rates are reported for the frozen ten-repository batch.Source: Table 20 · p. 32

D.6. Additional analyses

  • Additional analyses stratify results by repository and project characteristics, governance headroom, patch size, verification effort, and intervention locality.The public subset has one rollout per model-instance pair; about 110 paired observations, or roughly five rollouts per pair, are needed for 0.10 NGI resolution.Source: §D.6 · p. 32
  • Gate-off and gate-on settings preserve model ordering, while absent or vacuous gates cannot observe regressions, making their low breakage rates measurement artifacts.The gate-ablation analysis covers all instances available per model; the gate-strength analysis spans 60 instances and ten models.Source: Table 21 · p. 33, Table 22 · p. 33
  • Under no-op, five of ten instances drift; median NGI is 0.000 and standard deviation is 0.073.Dimension changes are measured on identical repositories.Source: Table 23 · p. 33

D.7. A corrected accounting bug

  • Correcting absent-gate accounting lowered affected models’ all-instance breakage rates from 15–17% to 6–7%; Qwen3.8-Max loses to a template on four of ten repositories.The public 22-instance subset was unaffected by the correction.Source: §D.7 · p. 33, Table 24 · p. 34

E. Reproducibility and Release

The release process separates public tasks, protected verification assets, results, and traces, while recording completeness and provenance. Its checklists prioritize regenerability, security review, and careful interpretation of comparisons.

E.1. Directory-level release layout

  • The release layout separates public tasks, protected verification assets, results, and traces, with a root manifest mapping tables to scripts and outputs to versioned inputs.Public artifacts support reconstruction of announced experiments without disclosing private outcomes or hidden answers.Source: §E.1 · p. 34, Table 25 · p. 34

E.2. Artifact completeness rules

  • An artifact counts as complete only when its manifest exists, its digest matches the stored file, and downstream tables identify whether it was used.Interrupted runs record the last completed stage and reason; an empty directory is not evidence of absence.Source: §E.2 · p. 34

E.3. Security and privacy

  • Licenses, credentials, private issue data, evaluator patches, and provider traces require separate release review; positive secret-scan results receive manual adjudication.Private tasks are excluded from public traces, score files, and aggregate tables.Source: §E.3 · p. 34

E.4. Result regeneration procedure

  • Regeneration scripts summarize machine-readable records, preserve the frozen manifest, distinguish valid from broken runs, and support a full-PDF visual check.Auxiliary settings and results are incorporated in the main text and Figure 4.Source: §E.4 · p. 35

E.5.1. Core identifiers

  • Each score record must reference an instance identifier for the repository snapshot, a run identifier for the model execution, and an evidence identifier for the probe.These identifiers remain stable across releases.Source: §E.5.1 · p. 35, Table 26 · p. 35

E.5.2. Researcher checklist

  • Before claiming auxiliary results, researchers should verify frozen repositories, shared base commits, protected evaluator assets, stable no-op results, visible denominators, and regenerable aggregates.Repeated seeds, same-scaffold reruns, and expert treatments are outside the primary comparison.Source: §E.5.2 · p. 35

E.5.3. Interpretation checklist

  • Compare paired coverage first, then behavior validity and per-dimension change, before aggregate NGI; reject apparent gains driven by smaller denominators, weaker gates, or more breakage.One instance can materially affect the mean in the ten-repository pilot.Source: §E.5.3 · p. 35
Loading 2609.29465v1…