PaperlayerView exact revision on arXiv

Source-linked AI summary

Superintelligent Software Engineering: Governance That Scales with Capability

Mohammad Mehdi Gholinejad

arXiv:2610.31306v1cs.SEcs.AI

TL;DR

As coding agents become more capable, project correctness still depends on decisions and contracts that code may not preserve, and that knowledge can become stale. The paper proposes a governance discipline for agents that outpace human review and examines agent context files in popular public repositories. Its snapshot finds such files are common and that many contain unresolved path references, while it reports no measured effect of its proposed discipline.

  • Problem

    Project correctness depends on decisions and contracts that may be absent from code and tests, while written project knowledge can become stale.

    Source: §I · p. 1, A. Correctness is relative to… · p. 3

  • Method

    The paper proposes governance that checks changes mechanically against project knowledge and checks that knowledge for decay before it is consulted.

    Source: C. Capability amplifies the rate of… · p. 4

  • Results

    In an exploratory snapshot of 452 eligible repositories, 220 (48.67%) had a root-level or .github context file, and 2,003 of 6,172 path candidates (32.45%) did not resolve.

    Source: C. Results · pp. 6–7

  • Takeaways & Limitations

    The paper argues that capability alone is insufficient and that governance must scale with it as agents outpace reviewers.

    Source: §XIII · p. 15

    The snapshot covers highly starred public repositories and does not generalize to GitHub as a whole or to private projects.

    Source: F. Threats to validity · p. 8

Abstract

from arXiv · show

Software agents are being trained to exceed human engineers; we argue that capability alone does not produce better software engineering. We define superintelligent software engineering as the discipline of building and evolving software with agents whose capability exceeds that of human engineers, in which governance (gates, contracts, decisions and decay control) scales with capability. The argument rests on three claims. First, correctness in a project rests on decisions and contracts that the code does not carry. Second, knowledge decays. Third, once an agent outpaces its reviewers, reading catches a shrinking share of the changes built on stale knowledge. A simple model shows that, past a fixed review capacity, escaping changes grow linearly with throughput while the share that reading catches falls toward zero; governance can lower the slope, capability alone cannot. An exploratory snapshot of 452 of the most-starred public GitHub repositories finds a root-level or .github agent context file in 48.67% (95% confidence interval 44.1 to 53.3); in the median file that names repository paths, 9.5% to 22% of those paths do not resolve, depending on how a reference is read. We then present OSTEON, a discipline for the regime, released with a DOI (10.5281/zenodo.23436020): each project gets a governed knowledge repository of decisions and contracts, every change is checked mechanically against it, and the knowledge itself is checked for decay at every session start; planning is gated, and sessions record what they learn without writing rules. We report no measured effect of OSTEON, mark claims about its mechanisms as opinion and the author's practice heuristics as design rationale, and state the questions that would falsify the thesis.

I. Introduction

As agents become more capable, project correctness still depends on decisions and contracts outside the code, and that knowledge can decay. The paper argues that governance must scale with capability and defines a discipline for the regime in which agents outpace human verification.

I. Introduction

  • Once throughput exceeds fixed review capacity, the model predicts that reading catches a shrinking share of stale-built changes while escapes grow linearly with throughput.The figure presents a model, not a measurement.Source: Figure 1 · p. 1
  • In 452 eligible public repositories, 48.67% had a root-level or .github agent context file, and the median file’s unresolved-path rate was 9.5% to 22%, depending on interpretation.An unresolved path is a tree mismatch, not evidence that an instruction is wrong.Source: §I · p. 1
  • Governance must scale in reach and fall in cost as agent capability grows, covering change gates, contracts, decisions, and knowledge decay.The paper argues that without this scaling, increased capability can produce more confident error.Source: §I · p. 1
  • The paper is a vision paper with one exploratory measurement; it reports no measured effect of OSTEON and treats the repository snapshot as observational, not causal.It also distinguishes the definition from the separate argument that the discipline is needed.Source: Evidence status · p. 2, §II · p. 2

II. Defining the Discipline

  • Superintelligent software engineering is defined as building and evolving software with agents beyond human engineers while governance scales with their capability.Its governance comprises gates, contracts, decisions, and decay control.Source: §II · p. 2
  • The relevant capability threshold is project-relative: an agent enters the regime when its output cannot be produced or verified line by line by project engineers in time.This can arise through depth, scope, or speed, rather than general superintelligence.Source: §II · p. 2

A. Three regimes

The paper distinguishes regimes by who can verify a change, then argues that the superseded regime requires verification to shift from reading changes to checking the knowledge and contracts behind them. Its starting point is that project correctness depends on intent that code and tests do not fully preserve.

A. Three regimes

  • In the superseded regime, throughput exceeds review capacity or reviewers cannot judge changes, so reading samples and verification shifts toward decisions and contracts.The discipline addresses this regime and its transition; projects can remain delegated by capping agents at review capacity.Source: A. Three regimes · pp. 2–3, Figure 2 · p. 3

B. Why name a discipline

  • The paper names a discipline for the regime in which agents outstrip reviewers, rather than an established research community or merely an era or agent type.It distinguishes its framing from Self-play SWE-RL [2], SE 3.0 [7], agentic software engineering [8], and the trust problem discussed by Roychoudhury et al. [9].Source: B. Why name a discipline · p. 3

III. Why Capability Is Not Enough

  • The argument that capability alone is insufficient rests on correctness, decay, and throughput, with further implications for oversight and authority.These claims motivate moving verification toward project knowledge when reading cannot keep pace.Source: §III · p. 3

A. Correctness is relative to knowledge the code does not carry

  • A change is correct relative to project intent, including decisions and contracts that may be absent from tests and unrecoverable from code.An agent cannot infer an unrecorded decision or know that an inferred decision has since been reversed.Source: A. Correctness is relative to… · p. 3

B. Project knowledge decays

Project knowledge can become stale, and context-file studies motivate keeping it current rather than simply adding more context. The paper’s model argues that stale-built changes escaping review grow linearly with throughput, while governance can reduce that growth by controlling staleness and checking every change.

B. Project knowledge decays

  • Stale documentation can mislead agents, and a controlled evaluation found repository context files generally did not improve task success while raising inference cost by over a fifth on average.The paper’s addition is that a small set of current, project-specific facts may be more useful than more context.Source: B. Project knowledge decays · p. 3

C. Capability amplifies the rate of stale-built change

  • The illustrative figure compares reading-only review with governance: decay control lowers s, and mechanical gates review every change.Its parameter values are illustrative, not measured.Source: Figure 3 · p. 4
  • Beyond review capacity c, stale-built changes escaping review grow linearly with throughput at the full stale fraction; below c, the slope is (1 − d) times that fraction.The threshold changes the slope, not the form of growth; for fixed s > 0, growth is unbounded. The model is not a measurement.Source: C. Capability amplifies the rate of… · pp. 3–4
  • Decay control lowers staleness s, while gates check every change and use detection probability g instead of readers’ d; a gate cannot catch stale, unflagged records.The gate’s detection probability is bounded by the currency of the record it checks, so decay control comes first.Source: C. Capability amplifies the rate of… · p. 4
  • Holding throughput at review capacity or adding reviewers are alternatives, but each gives up capability or increases review capacity rather than improving automatically with the agent.The paper’s scaling claim is that governance, not capability alone, must improve with the agent.Source: C. Capability amplifies the rate of… · p. 4

D. Verification moves from the change to the knowledge

When reviewers cannot judge every change, the paper proposes shifting human verification to the decisions and contracts that changes are checked against. Its scaling principle combines low-cost, broad, current checks with human authority over decisions and learning that does not let sessions write rules.

D. Verification moves from the change to the knowledge

  • Human verification can focus on short, stable, human-approved decisions and contracts, which are fewer than changes; the change-to-knowledge step is mechanized.This is the paper’s response to reviewers being unable to judge every change.Source: D. Verification moves from the change… · p. 4

A decision records

  • Project correctness depends on trade-offs, neighboring contracts, and system state that code alone does not carry.The paper locates these decisions with accountable project owners, not agents.Source: A decision records · p. 4

F. The scaling principle

  • Governance scales when cost per change falls, governed scope expands to durable choices and seams, and provenance-backed decay checks occur before work starts.The paper treats cost and coverage as scaling properties and currency as an invariant.Source: F. The scaling principle · p. 5
  • Agents propose changes to decisions or contracts, but accountable humans make the final call; human effort attaches to rare decisions rather than numerous changes.Source: F. The scaling principle · p. 5
  • Sessions may improve the process from what they learn, but they do not legislate rules for later sessions.Source: F. The scaling principle · p. 5

IV. Context Decay in Public Repositories: An Exploratory Measurement

  • The exploratory observational snapshot asks about context-file prevalence, currency, and path resolution, and makes no causal claim.It examines agent context files in public repositories as a test of the premise that written project knowledge drifts.Source: §IV · p. 5

A. Method

The study sampled popular public repositories using a pre-specified frame and exact list of root-level or .github context-file names. It measured file age and update activity, plus whether path-like references resolved against each repository tree.

A. Method

  • The frame comprised the first 500 repositories returned for stars:>10000, fork:false, archived:false, sorted by stars; 452 eligible repositories were measured.The query returned 5,346 repositories; exclusions included 47 without a primary language and one confidentially withheld repository.Source: A. Method · p. 5
  • The sample included exact, case-sensitive matches for four root-level filenames and .github/copilot-instructions.md; nested AGENTS.md and CLAUDE.md files were counted but not analysed.Each qualifying context file was one unit of analysis.Source: A. Method · p. 5
  • The frame, filenames, metrics, and summary statistics were fixed before collection, while sensitivity analyses and other comparisons added later were labelled post hoc.The pre-specification was documented in released files, not registered with a third party.Source: A. Method · p. 5

B. Metrics

  • File age and days since update measure time from the oldest and newest touching commits, while updates count commits touching the path.Path history does not follow renames, so a moved file is aged from its move.Source: B. Metrics · p. 5
  • Commits since update counts later default-branch commits, while dangling-reference rate is the share of distinct path-like candidates that fail to resolve in the snapshot tree.Candidates resolve as exact paths or directory prefixes, either from the repository root or relative to the context file; unresolved paths may reflect drift, planned or generated paths, or extraction heuristics.Source: B. Metrics · p. 6

C. Results

The snapshot found agent context files in 220 of 452 eligible repositories, and many files with path-like references had unresolved candidates. Currency measures and sensitivity readings qualify how those descriptive findings should be interpreted.

C. Results

  • 48.67% of eligible repositories (220/452; 95% CI 44.1–53.3) carried at least one root-level or .github context file.The post hoc language breakdown ranged from 62/85 TypeScript and 23/32 Go repositories to 4/15 Java repositories.Source: C. Results · p. 6
  • 220 days was the median context-file age, and repositories had a median of 229 commits since the file’s last update.Median days since update was 48; Fig. 4 plots commits since update against days since update, with markers shaded by dangling-reference rate when candidates exist.Source: C. Results · p. 6, Figure 4 · p. 6
  • 22% was the median dangling-reference rate among 269 files with path-like candidates; 71.38% had at least one unresolved reference.The pooled rate was 2,003 of 6,172 candidates (32.45%); Fig. 5 shows the distribution and its median.Source: C. Results · pp. 6–7, Figure 5 · p. 7
  • The file-level association between commits since update and dangling-reference rate was not positive (Spearman rS = −0.15, n = 269).The authors report no causal interpretation; repositories differ in size, activity, and how their files refer to paths.Source: C. Results · p. 7

D. Sensitivity of the dangling-reference rate (post hoc)

  • Bare extensions and conventionally untracked build or local paths accounted for 9.59% of unresolved references in the post hoc audit.The remaining candidates also include bare filenames that may exist deeper in the tree but are not resolved by the protocol.Source: D. Sensitivity of the… · p. 7
  • The median dangling-reference rate fell from 0.2222 under the pre-specified reading to 0.1818 under reading A and 0.0952 under reading B.The share of files with any unresolved reference also fell from 71.38% to 68.28% and 61.44%; the headline rate depends strongly on how references are read.Source: D. Sensitivity of the… · p. 7, Table III · p. 7

E. What the measurement shows, and what it does not

The snapshot finds unresolved path references to be common but does not establish how they arose or what they cause. The section also sets validity boundaries and presents seven proposed requirements for a discipline.

E. What the measurement shows, and what it does not

  • Most measured files naming repository paths included at least one unresolved reference, but the snapshot does not show how the mismatch arose or what it causes.Files unchanged through more commits did not have more unresolved references at file or repository level.Source: E. What the measurement shows, and… · p. 7

F. Threats to validity

  • Pattern matching can flag strings that are not paths and miss real references, so the unresolved-reference measure is a symptom rather than a defect.The post hoc readings bound only part of this effect; the snapshot cannot separate path drift from false positives that increase with file length.Source: F. Threats to validity · pp. 7–8
  • File-level observations are not independent within repositories, and nested-file counts for seven repositories with truncated trees are lower bounds.Repository-level figures serve as a check on the file-level analysis.Source: F. Threats to validity · p. 8
  • The findings describe highly starred public repositories above 50,000 stars and do not generalize to GitHub as a whole or private projects.Search ranking and repository contents change, although the released scripts and per-file data permit recomputation.Source: F. Threats to validity · p. 8

V. Requirements for a Discipline

  • The authors propose seven requirements for a discipline, covering fact ownership, recorded decisions, seam contracts, project-specific gates, decay checks, gated planning, and learning without self-legislation.The requirements were written alongside OSTEON, not derived independently of its design, and should be tested against other practices.Source: §V · p. 8

VI. OSTEON

OSTEON is a proposed discipline for governing software work through a tiered project knowledge repository, recorded decisions, and human-approved curation. Its mechanisms connect knowledge maintenance with planned, reviewed change.

VI. OSTEON

  • Release 0.1.0 is a Claude Code plugin with fifteen skills, a scaffold for the project knowledge repository, and a session-start decay check.The release is proposed as a discipline, not reported as a measured intervention.Source: §VI · p. 8

A. Tiers and the BONE

  • OSTEON assigns facts to three tiers and places each at the narrowest tier that fully owns it.The project tier, or bone, is a dedicated repository with four knowledge layers and a manifest; agents and humans both read it, and changes are gated before merging.Source: A. Tiers and the BONE · pp. 8–9, Figure 6 · p. 9
  • The bone gives project knowledge repository history, branches, review, and provenance, while separating project facts from repository-specific conventions.A fact’s tier depends on what would change it, not merely its topic.Source: A. Tiers and the BONE · p. 9

B. Decision records

  • Decision records have proposed, accepted, and superseded states; accepted records are amended or superseded rather than silently rewritten.This makes a change that suspends a decision or conflicts with the running system explicit in the record.Source: B. Decision records · p. 9

C. Module contracts

OSTEON links written module contracts and project-specific review to decay checks, curation, and gated planning. The design assigns mechanical checks to agents while reserving approval of knowledge changes for humans.

C. Module contracts

  • Every module seam has a versioned contract, and a contract change ships in the same change request as the interface change it describes.The contract skill also flags work built against another module’s source code instead of its written contract.Source: C. Module contracts · p. 9

D. Review gates and curation

  • Review checks changes against the project’s own decisions, contracts, and rules, then routes detected invalidations to curation rather than editing the knowledge base.Curation proposes knowledge updates but applies nothing without per-item human approval.Source: D. Review gates and curation · pp. 9–10

E. The session-start decay check

  • At session start, the decay check compares knowledge-unit provenance with a sealed baseline and reports units whose cited sources have changed.The index is an exposure measure, not confirmation that a decayed unit is wrong; curation decides what to do.Source: E. The session-start decay check · p. 10

F. Ossification: planned and gated change

  • Ossification gates planned work before execution, checks plans against ten conditions, and treats a merge as incomplete until the plan records completion.The planner maintains a dated cycle and a definition of done for each task.Source: F. Ossification: planned and gated… · p. 10

G. Capture without self-legislation

OSTEON records session learning without letting sessions make governing rules, and its own history shows only partial use of its mechanisms. The paper relates the design to governance axes and illustrates it with constructed scenarios, not observations.

G. Capture without self-legislation

  • Sessions record observations in a non-governing ledger; a rule requires independent corroboration and human approval before entering the register.The capture skill nominates corroborated observations and stops short of making them rules.Source: G. Capture without self-legislation · p. 10

H. What OSTEON’s own history shows

  • OSTEON’s six-repository, 58-commit history shows partial mechanism use by one author, not evidence of effectiveness.Capture ran over three days and produced three nominations and one curation record; no nomination became a rule, and the tiers, gates, and decay check were not applied to OSTEON’s own development.Source: H. What OSTEON’s own history shows · p. 11

VII. How Governance Scales in OSTEON

  • The scaling account proposes mechanical checks for cost and coverage, provenance-based decay detection for currency, human approval for authority, and corroboration for learning.Whether block-level provenance keeps warnings per change falling is an open question in the paper’s research agenda.Source: §VII · p. 11

A. Constructed scenarios

  • Three constructed scenarios illustrate governance responses to a drifting seam, a silently suspended decision, and a stale state record.In each scenario, the authors say governance acts on the knowledge the change depended on; the scenarios are not observations.Source: A. Constructed scenarios · p. 11

VIII. Design Rationale from Practice

The practice heuristics are design rationale, not findings; the objections and related-work discussion distinguish OSTEON’s governance approach from adjacent practices. The paper frames its novelty as combining established mechanisms into a discipline for projects where reviewers can no longer verify changes by reading.

VIII. Design Rationale from Practice

  • The author’s practice heuristics are explicitly design rationale, not findings, and the paper reports no project data or episodes supporting them.Their corresponding OSTEON mechanisms do not corroborate the heuristics.Source: §VIII · pp. 11–12

IX. Objections

  • Provenance is required for agents to know whether documentation is stale; tests cover exercised behavior but omit decisions, ownership across seams, and production state.A change can pass every test and still reverse a decision.Source: §IX · p. 12
  • Governance charged to humans per change may be skipped when changes outpace their available time; mechanical per-change checks are the proposed response, not a demonstrated OSTEON result.Whether OSTEON achieves this is an open question.Source: §IX · p. 12
  • OSTEON’s stated novelties are per-block decay checks against provenance and sealed source baselines, a fixed human-authority boundary, and a discipline for limited review capacity.The paper distinguishes this from documentation-as-code practices and inconsistency checks for knowledge inside code.Source: §IX · p. 12

X. Related Work

  • Compared with SE 3.0 [7] and agentic software engineering [8], OSTEON defines the regime by verification capacity and makes governance scaling, rather than partnership, its defining property.The paper describes its start gate and evidence-backed review as resembling consultation requests and merge-readiness packs [8].Source: §X · p. 12
  • The training route uses test patches to specify bugs [2], while OSTEON addresses decisions, contracts, and running-system state that training does not see.Context-file studies report mixed cost findings [16], [26]; this paper measures path resolution, not effects on agents, and treats transfer to governed knowledge as open.Source: §X · pp. 12–13
  • ContextNest governs knowledge eligibility and integrity [27], while OSTEON targets project decisions, contracts, decay, and planned change; the paper says the approaches can compose.Deliberative Curation [28] may inform how items reach human approval as agent counts grow, an open design question.Source: §X · p. 13
  • Prior work studies documentation drift and traceability,,,,,,; OSTEON instead checks knowledge absent from code against provenance and a sealed source baseline.The paper also grounds OSTEON in decision records, contracts, provenance, and software-evolution work, combining these into a discipline with decay checked at session start,,,,,,,,,.Source: §X · p. 13

XI. Research Agenda

The research agenda sets out six questions that could test or bound the thesis, from whether governed knowledge reduces stale-built changes to when agents may curate. It specifies shared experimental conditions, including released instruments and clean decay runs.

XI. Research Agenda

  • Q1 tests whether governed knowledge reduces changes contradicting fixed decisions and contracts, while varying throughput at fixed review capacity; no arm difference would weaken the thesis.The experiment uses experimenter-defined decisions and contracts as the oracle in both arms.Source: §XI · p. 14
  • Q2 measures human governance minutes against change counts; proportional growth in human effort, or reduced effort from skipped governance, would fail the scaling principle.Source: §XI · p. 14
  • Q3 measures decay time for anchored knowledge and the share of warnings leading to curation; near-zero decay would make R5 unnecessary.Section IV’s snapshot is a coarse reference point, not a decay rate; governed-knowledge decay remains open.Source: §XI · p. 14
  • Q4 compares context files with governed units when a recorded decision is current or superseded; no difference in the superseded condition would undermine the claimed distinction.The two conditions use the same text when current, while only the governed unit is flagged as decayed after supersession.Source: §XI · p. 14
  • Q5 tests whether gates still detect contradictions as agent capability rises, including when agents argue persuasively; falling detection would make escape rates depend on the agent again.Source: §XI · p. 14
  • Q6 compares per-item human approval with deliberative multi-agent curation on error rate and human cost to price the authority boundary without moving it.It is an open design question, not a test of the thesis.Source: §XI · p. 14
  • All six questions call for released instruments when project data cannot be shared and decay series that include clean runs for rate denominators.Source: §XI · p. 14

XII. Limitations

The paper reports no measured effect of OSTEON and bounds its exploratory repository snapshot and model. It concludes that OSTEON is a concrete proposed discipline whose effectiveness remains an empirical question.

XII. Limitations

  • The paper reports no measurement of OSTEON’s effect, so it does not show that the discipline works.Source: §XII · p. 14
  • The repository study is one observational snapshot of popular public repositories; unresolved paths are only a possible symptom of decay, and the headline rate depends on how references are read.It does not measure wrong instructions or their effects on agents.Source: §XII · p. 15
  • The escape model assumes independent staleness, fixed review capacity and detection probability, and fixed staleness as throughput rises; the paper presents its direction, not its size.The authors note that agent changes can make other units stale, so holding staleness fixed understates the effect in that respect.Source: §XII · p. 15
  • The capability threshold is imprecise and may be crossed at different times for different tasks; OSTEON 0.1.0 targets one agent platform and was designed by one author.Independent use may expose costs not yet seen.Source: §XII · p. 15
  • The authors argue governance may reasonably be refused while agents remain in the delegated regime; their claim concerns the superseded regime, where they argue the alternative is worse.Source: §XII · p. 15

XIII. Conclusion

  • The conclusion presents OSTEON as one concrete discipline and says whether it works is an empirical question; its release includes scripts and data.OSTEON 0.1.0 is archived on Zenodo under DOI 10.5281/zenodo.23436020.Source: §XIII · p. 15, Availability · p. 15

Related papers

Loading 2610.31306v1…