PaperlayerView exact revision on arXiv

Source-linked AI summary

Relic: From Multi-Agent Collaboration to Persistent Organizational Capability

Hongyi Du, Tianyi Zhang, Weijia Zhang, Yi Yang, Haofei Yu, Kunlun Zhu, Tianxiang Dai, Shang Jiang, Zhelun Gao, Jiaxin Pei, Shang Zhu, Jiaxuan You

arXiv:2609.32965v1cs.AIcs.SE

TL;DR

Multi-agent work can break when members act on stale shared artifacts, and agreements may not carry over when participants change. Relic makes collaboration-derived rules governed and executable beyond member-local histories; across controlled software runs, it improves complete-contract delivery over a matched structured team.

  • Problem

    Shared software work can break when agents use stale interfaces or tests, and conversational agreements do not automatically govern replacement members.

    Source: §1 · p. 1

  • Method

    Relic turns visible collaboration friction into organization-owned rules that undergo validation and approval before runtime binding, and can be revised or retired.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    5.71 pp: Relic improves pooled complete contracts over the matched structured team across 360 controlled runs, with gains on all four verified endpoints in every model stratum.

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    The results support persistent executable organizational state as a useful layer for long-horizon, persistent multi-agent collaboration.

    Source: §7 · p. 10

    The evaluation focuses on software-production settings; broader domains remain for future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Multiple agents may often conflict in an organization: for example, one coding agent changes an interface in a repository, but another continues to develop on the old version where existing tests become stale. A conversation can resolve the episode, but when the participants change, what makes the lesson continue to govern the team? We introduce Relic, which turns recurring collaboration failures into organization-owned, executable protocols. Members reflect on visible work, propose rules, and govern their adoption. Adopted protocols bind triggers, responsibilities, required evidence, and execution consequences to the runtime, while remaining open to revision and retirement. In one traced case, repeated integration friction produces an interface-review rule that governs later pull requests and is revised as work continues. Across 360 controlled runs over ten software workloads and three models, Relic raises complete-contract delivery from 14.06% to 19.76% (+5.71 percentage points) over a matched structured team without the protocol lifecycle, improving all four verified production endpoints in every model stratum. Under fresh-member transfer, behavioral correctness is 25.4% with no inherited protocol, 34.6% with the same rules provided as readable text, and 41.2% with executable bindings, a +6.5-point advantage over text alone. On the full CooperBench benchmark, after excluding broken benchmark pairs, Relic achieves 367/477 (76.9%), establishing the best reported result among peer-structured systems. On the fixed 48-pair same-model subset, Relic also exceeds Solo (29/48 vs. 26/48), reversing the coordination loss exhibited by the official peer baseline. Together, these results show how collaboration experience can become persistent organizational state that remains useful beyond the members who created it.

1 INTRODUCTION

Shared software work can fail when agents act on stale interfaces, tests, or branches, and agreements made in conversation do not automatically govern replacement members. Relic turns observed collaboration friction into governed, executable organizational rules that persist beyond their authors.

The continuity problem

  • An interface change can leave a client using the old interface, while lagging branches, overlapping edits, and outdated tests further complicate integration.Shared work therefore needs rules for artifact access, synchronization, and integration evidence.Source: §1 · p. 1
  • Even perfect agreement to synchronize, rerun tests, and obtain review does not automatically bind a replacement agent to that agreement.Fresh sessions may lack prior memory, and structured handoffs add orchestration overhead.Source: §1 · pp. 1–2

Why skills are not enough

  • A testing skill can run a suite, but its availability alone assigns no independent reviewer or current-evidence requirement before integration.Relic binds learned rules to shared work by specifying when they apply, who is responsible, and how compliance and revisions are governed.Source: §1 · p. 2

Contributions

  • Relic represents learned ways of working as organization-owned, governed executable protocols outside member-local state.Members propose rules from visible work friction; validation and approval precede runtime binding, and rules remain revisable.Source: §1 · p. 2
  • Across 360 runs, Relic improves pooled complete contracts by 5.71 pp over the matched structured team, with gains on all four verified endpoints in every model stratum.The CooperBench result is 367/477 (76.9%) after excluding broken benchmark pairs, the best reported result among peer-structured systems.Source: §1 · p. 2

2 RELATED WORK

Prior work studies agent communication and workflows, persistent memory and skills, adaptive agent organizations, and collective routines. Relic focuses on making collaboration-derived rules explicit, governed, executable, and reusable across members.

Communication and shared workflows

Memory and reusable skills

  • Memory systems retain feedback, extracted experience, and collaboration insights, while Voyager accumulates executable skills; Relic instead binds obligations and evidence responsibilities to shared work (Shinn et al., 2023; Zhao et al., 2023; Wang et al., 2023).A skill supplies a testing procedure, whereas a protocol assigns responsibility for producing and reviewing required evidence.Source: §2 · p. 3

Self-evolving agent organizations

  • Adaptive organizations learn agent behavior, team structures, or recruitment, while Relic focuses on evidence-grounded protocol proposal, adoption, execution, and revision (Hao et al., 2026; Yu et al., 2026; Lyu et al., 2026).Its rule content evolves while model parameters and the decision and execution substrate remain fixed.Source: §2 · p. 3

Organizational routines and governance

  • Organizational routines and CSCW study retained or reconfigured practices and coordination of interdependent tasks; Relic makes learned rules explicit, revisable objects outside member-local state.Its evaluation separately measures sustained practice, verified outcomes, and usefulness after member replacement.Source: §2 · p. 3

3 RELIC: FROM EXPERIENCE TO ORGANIZATIONAL PROTOCOLS

Relic turns observed work friction into organization-owned protocols through governed adoption, then binds those protocols to decisions and execution. The rules specify obligations for shared work and can continue to govern later role occupants.

3 RELIC: FROM EXPERIENCE TO ORGANIZATIONAL PROTOCOLS

  • Relic adds a governed protocol lifecycle in which members propose rules from observed problems and validation and approval precede runtime binding, revision, or retirement.Figure 2 traces an interface-review protocol from its origin through execution and reuse by a new member.Source: §3 · p. 3

3.1 WHAT AN ORGANIZATIONAL PROTOCOL CONTAINS

  • Protocols are distinct from member-local and shared-work state, and specify triggers, scope, responsible roles, steps, evidence, affected actions, consequences, and revision or retirement state.Members see only information they access and read or retrieve; evaluator audit records support measurement and replay separately from member context.Source: §3.1 · p. 3

3.2 FROM WORK FRICTION TO GOVERNED PROTOCOLS

  • Repeated interface failures can prompt evidence-grounded proposals; validation checks evidence and compatibility, approval adopts a rule, and amendments repeat governance while retirement disables the rule but preserves its history.Protocol content changes online while model parameters and validation and execution code remain fixed; sustained practice is measured through adoption followed by repeated use across independent units and times.Source: §3.2 · p. 4

3.3 HOW PROTOCOLS GOVERN LATER WORK

  • SDL scores feasible actions from visible work, member, and applicable protocol features, so protocol obligations can alter action priorities; role mappings route responsibility and execution checks update or reject work transitions.The scoring map, action registry, and handlers remain fixed as protocols evolve.Source: §3.3 · p. 4

3.4 MEMBER-INDEPENDENT REUSE

  • A protocol remains with a role when its member is replaced, and the transfer experiment gives fresh members designated protocols and role mappings without source code, world state, or private histories.Text provides frozen rules as prose, while Exec adds runtime bindings; B3 enables the governed protocol lifecycle on the shared control backbone.Source: §3.4 · pp. 4–5, Figure 2 · p. 5

4 EVALUATION: FORMATION, EFFECTIVENESS, AND TRANSFER

The evaluation tests protocol formation, verified software delivery, executable binding, transfer, and portability across controlled workloads and external benchmarks. Its designs compare Relic with a structured control, a prose-only variant, and content-matched transfer conditions.

Workloads and outcomes

  • Ten workloads comprise five program-building tasks and five repository repair or extension tasks with interdependent implementation, testing, review, and integration.A contract is complete only when every acceptance case passes on mainline; exposed and held-out cases test visible and withheld requirements, and private tests and reference implementations remain hidden.Source: §4 · p. 5

Conditions and design

  • Three models each run ten workloads with three seeds under B0–B3, totaling 360 controlled runs of 336 scheduling steps.Relic B3 and structured control B2 share model, tools, task-visible information, SDL parameters, member learning, and execution code, isolating the governed protocol lifecycle comparison.Source: §4 · p. 5

Binding, transfer, and external evidence

  • A 30-run Claude Opus 4.6 ablation compares B3 with B3-text, which retains rule proposal, governance, adoption, revision, and readable rules but removes executable bindings.The comparison is designed to test binding separately from those lifecycle components.Source: §4 · p. 5, Figure 3 · p. 6
  • Content-matched Text and Exec transfer tests use fresh members and the same frozen protocols, with only Exec receiving bindings; ProgramBench tests portability without SDL and CooperBench tests two-agent interacting-feature delivery.CooperBench comparisons include the full benchmark and a fixed 48-pair same-model Solo–Peer subset.Source: §4 · p. 6, Figure 3 · p. 6

5 RESULTS

Relic improves verified delivery over a matched structured control, while its learned protocols govern work allocation and transfer to fresh members. The results also extend to external software benchmarks.

5.1 EXECUTABLE PROTOCOLS IMPROVE VERIFIED DELIVERY

  • Complete contracts rise from 14.06% to 19.76% (+5.71 pp) with Relic over the structured control, and all four verified endpoints improve in every model stratum.Table 1 reports the pooled three-model, 360-run result; the other endpoint gains are +7.72 pp for held-out, +6.58 pp for exposed, and +6.96 pp for confirmed issues.Source: Table 1 · p. 6, §5.1 · p. 6

5.2 WHAT PROTOCOLS DOES THE ORGANIZATION LEARN?

  • Across the Terra+Opus 60-run census, 280 of 497 protocol lineages meet the sustained-use criterion, with most formed rules concerning review/merge, release engineering, and evidence governance.Thirty-two lineages additionally link other members’ execution to shared-state changes and independently evaluated outcomes.Source: §5.2 · p. 7
  • Relic moves 36% of accepted work to mainline versus 32% for the structured control, while its PR merge rate is 50% versus 44%.The single agent has the highest patch-acceptance rate (86%) but the lowest complete-contract score, showing that local acceptance alone does not ensure delivery.Source: §5.2 · p. 7, Figure 5 · p. 7
  • Relic averages 7.259M tokens per run versus 4.274M for the control; matched-spend sensitivity estimates remain positive on all four verified endpoints.The paired B3–B2 tokens-per-issue estimate is −0.448M (95% CI [−1.573M, +0.659M]); matched-spend complete-contract gain is +7.45 pp.Source: §5.2 · pp. 7–8

5.3 EXECUTABLE BINDING AND MEMBER-INDEPENDENT TRANSFER

  • With fresh members, executable bindings pass 41.2% of behavioral cases, versus 34.6% for the same rules as text and 25.4% with no inherited rules.Exec exceeds Text by 6.5 pp (95% CI [0.7, 15.6]).Source: §5.3 · p. 8, Table 2 · p. 8
  • In a separate 30-run Opus binding ablation, complete contracts fall from 22.20% to 15.03% when learned-protocol bindings are removed.Both conditions develop rules independently; unlike the content-matched transfer test, this ablation disables bindings during rule creation.Source: §5.3 · p. 8

5.4 EXTERNAL SOFTWARE BENCHMARK EXTENSIONS

  • On CooperBench, Relic scores 367/477 (76.9%), exceeding the released peer reference at 282/477 (59.1%) and hierarchical Team reference at 354/477 (74.2%).On the fixed 48-pair same-model subset, Relic reaches 29/48 versus Solo’s 26/48, while Official Peer reaches 13/48.Source: §5.4 · p. 9, Table 3 · p. 9
  • On 25 ProgramBench tasks, adding a frozen package of executable and advisory rules raises mini-SWE-agent’s mean behavioral pass rate from 64.164% to 70.916% (+6.752 pp).The package covers originality, evidence freshness, verification, and submission readiness.Source: §5.4 · p. 9

6 CASE STUDY: FROM REPEATED FRICTION TO A GOVERNING RULE

A matched case traces repeated integration friction into an interface-review protocol that is adopted, enforced, and amended as later work proceeds.

Before: activity without sustained delivery

  • Before the protocol, the role-based team accepted 168 patches but merged only two PRs, while Relic accepted 118 and merged 76.The case holds model, random seed, tools, and horizon fixed.Source: §6 · p. 9

Formation: repeated friction becomes a rule

  • Repeated interface and verification failures produced a rule requiring changed public interfaces to match the written contract and have current test evidence before approval.Source: §6 · p. 10

After: the rule governs later work

  • Proposed at step 12 and adopted at 21, the rule was first enforced at 26, amended at 33 and 59, and used 419 times through step 336.It recorded 211 enforcements across 12 PRs; 11 affected PRs later merged, including PR-A after its initial block.Source: Figure 6 · p. 9, §6 · p. 10

7 DISCUSSION, FUTURE WORK, AND CONCLUSION

The paper frames Relic as revisable organizational state that persists beyond member-local histories and supports long-horizon multi-agent collaboration. Its evaluation focuses on software production, with broader domains and governance questions left for future work.

7 DISCUSSION, FUTURE WORK, AND CONCLUSION

  • The discussion presents Relic as shared rules formed from experience, executed, revised, and reused by later members, with benefits recurring across model strata and benchmark settings.The conclusion characterizes persistent executable organizational state as a useful layer for long-horizon collaboration.Source: §7 · p. 10
  • The evaluation focuses on software-production settings; broader domains, selective transfer and forgetting, rule-quality diagnosis, alternative decision layers, and human–organization interaction remain future directions.Source: §7 · p. 10

ETHICS STATEMENT

  • The reported human-centered work consists of formative paired interface walkthroughs by four author-team members uninvolved in Relic’s design or implementation.Their interaction experience and feedback were reported as formative internal observations.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The appendices document experimental conditions, workload construction, configurations, seeds, procedures, metric definitions, statistical methods, evaluator isolation, checkpointing, and recovery policies to support reproduction and inspection.Source: REPRODUCIBILITY STATEMENT · p. 11

AI USE STATEMENT

  • Generative AI tools assisted across benchmark construction, design, implementation, analysis, and drafting; the authors reviewed artifacts and analyses, verified results, and made final research decisions.Source: AI USE STATEMENT · p. 11

A.1 EVIDENCE MAP: WHAT EACH EXPERIMENT ESTABLISHES

The appendix frames organizational capability as a persistent systems layer distinct from individual capability, with protocols governing how members coordinate and deliver shared work. It highlights unresolved challenges in rule governance and evaluation beyond software settings.

A.1 EVIDENCE MAP: WHAT EACH EXPERIMENT ESTABLISHES

  • Table 4 organizes experimental results across formation, binding, transfer, cost, and external benchmarks.Source: Table 4 · p. 16

A.2 ORGANIZATIONAL CAPABILITY AS A SYSTEMS LAYER

  • Relic separates member capability from organizational state that routes evidence, assigns responsibility, reviews work, and turns local progress into shared delivery.In this software-production setting, observed protocol lineages concentrate near the boundary between local work and trustworthy integrated delivery.Source: §A.2 · p. 16

A.3 AUTONOMOUS FORMATION OR FIXED PROTOCOL PACKAGES?

  • B3 develops rules through proposal, governance, execution, revision, and retirement, whereas Transfer Exec deploys a frozen package to fresh members.These represent adapting rules during work and reusing an established package at initialization.Source: §A.3 · p. 16

A.4 THE ORGANIZATIONAL MATURITY GAP

  • The paper describes an organizational maturity gap: individual agent capability advances faster than persistent structures for collective work.Relic studies executable protocols as durable work rules specifying responsibility, evidence, review, integration, exceptions, and revision; other organizational objects may also support persistent capability.Source: §A.4 · pp. 16–17

A.5 GOVERNANCE TRADES COST FOR RELIABILITY

  • Governance routes responsibility, review, and verification through shared procedures, while B3 incurs more tokens per run and retains positive endpoint gains under matched-spend analysis.Source: §A.5 · p. 17

A.6 STRONG EXECUTION RAISES THE STAKES OF RULE QUALITY

  • Protocol triggers, evidence requirements, responsibilities, and revision paths shape how experience governs later work, making inspection, amendment, retirement, and reuse explicit.Source: §A.6 · p. 17

A.7 FROM RULE FORMATION TO EVIDENCE-BASED META-GOVERNANCE

  • Larger organizations must govern who proposes, approves, overrides, revises, or retires rules, their scope, conflict resolution, and legitimate exceptions.The text proposes evidence-based deployment through limited-scope trials, explicit outcomes, staged expansion, and revision or sunset conditions.Source: §A.7 · p. 17

B RESEARCH AGENDA FOR PERSISTENT AI ORGANIZATIONS

B.2 ORGANIZATIONS AS A FIRST-CLASS EVALUATION UNIT

The paper proposes evaluating persistent AI systems at the organizational level and outlines research directions for measuring capability, transfer, governance, and human interaction. It also identifies cross-organizational structures as a further scale of study.

B.2 ORGANIZATIONS AS A FIRST-CLASS EVALUATION UNIT

  • Organizational-level benchmarks could vary topology, hierarchy, authority, roles, turnover, and memory while holding the work environment fixed.Such studies could expose organization-level failure modes that task-bounded benchmarks cannot express.Source: §B.2 · p. 17

B.3 CAPABILITY AND CAPACITY TAXONOMIES

  • Capability evaluation should separate model, agent, temporary-team, and persistent organizational capability, while capacity evaluation asks how much work and complexity each sustains.The proposed dimensions include formation, scope, executability, persistence, transferability, compositionality, and reversibility.Source: §B.3 · pp. 17–18

B.4 TRANSFER AND SELECTIVE FORGETTING OF ORGANIZATIONAL STATE

  • Transfer studies could test partial or gradual turnover, cross-model and cross-domain transfer, interference, and when shared state should be adapted or forgotten.Candidate transferred state includes protocols, workflows, roles, and responsibility maps.Source: §B.4 · p. 18

B.5 CAPABILITY ECOLOGY AND FAILURES OF ORGANIZATIONAL LEARNING

  • Research could trace how capability families compose, compete, become obsolete, or depend on one another, and measure failures to institutionalize recurring friction.A formation funnel could track friction exposure through recognition, proposal, adoption, formation, and strong-effect evidence.Source: §B.5 · p. 18

B.6 HUMAN–AGENT ORGANIZATION INTERACTION

  • Human–agent studies could compare direct participation with secretary-mediated interaction, including how organizational interfaces handle explanation, approval, delegation, and accountability.The proposed alternatives include a secretary that summarizes state, surfaces decisions, and translates human intent into organizational actions.Source: §B.6 · p. 18

B.7 TOWARD AGENTIC SOCIOLOGY

  • Agentic sociology frames persistent roles, norms, institutions, and inter-organizational relations as a research scale between isolated interactions and multi-organization systems.Communication interoperability alone does not establish task ownership, evidence obligations, revision authority, or dispute resolution.Source: §B.7 · p. 18

C EXTENDED METHOD AND FORMAL SPECIFICATION

The extended specification separates member-local, shared-work, and protocol state, then describes how governed rules can persist, affect decisions, and be measured in a software-production environment. It also details the controlled workloads and execution conditions used in the study.

C.1.2 ORGANIZATIONAL STATE

  • Organizational state combines shared work objects, responsibility and decision-right mappings, and adopted protocols, with a rule operationally distinguished by an active runtime binding.Member-local memory and experience can affect one member’s decisions without becoming organization-owned state.Source: §C.1.2 · p. 19, §C.1.1 · p. 19

C.2 PERSISTENT ORGANIZATIONS AND ORGANIZATIONAL LEARNING

  • A persistent organization stores learned rules and cross-member responsibilities outside member-local histories, while organizational learning changes governed mechanisms that can affect members who did not create them.Replacement resets private state while shared work can persist; the transfer experiment specifies the protocols and responsibility mappings given to a fresh target.Source: §C.2 · p. 20, §C.2.1 · p. 20, §C.2.2 · p. 20

C.3 STRUCTURED DECISION LAYER AND EXECUTION

  • The structured decision layer constructs feasible candidates from visible work and role constraints, then selects actions; the common execution layer validates transitions and applies protocol-linked checks.Candidate construction is shared across conditions, while action selection differs; action handlers and execution code remain fixed as protocols evolve.Source: §C.3 · p. 20, §C.3.1 · p. 21, §C.3.2 · p. 22

C.4 CAPABILITY FORMATION AND GOVERNANCE

  • Recurring friction can produce a proposal that is checked for evidence grounding and runtime compatibility, governed for adoption, compiled into organizational state, and later amended or retired.Retirement stops future use while retaining lifecycle history; capability formation is measured through repeated use over distinct units and times, with stronger evidence linking third-party enforcement to work changes and evaluator evidence.Source: §C.4.2 · p. 22

D EXECUTION-GROUNDED ORGANIZATIONAL ENVIRONMENT

  • The shared software environment uses persistent work objects, partial observability, executable transitions, and a frozen evaluator, with private, shared, and evaluator-only information separated.Product progress is measured through concrete state transitions, and evaluator-only tests and held-out requirements remain invisible to members.Source: D EXECUTION-GROUNDED ORGANIZATIONAL… · p. 23, §D.1 · p. 23, §D.2 · p. 24

D.4 SOFTWARE-PRODUCTION LIFECYCLE AND VERIFICATION

  • Repository delivery follows edit, commit, pull request, review or CI, and merge; new commits invalidate earlier CI evidence for that pull-request head.The main-study horizon is 336 scheduling steps with checkpoints every 24 steps, and members take at most one primary action per step.Source: §D.4 · p. 25, §D.5 · p. 25

E BENCHMARK CURATION

  • The ten frozen workloads span repository repair or feature development and zero-to-one construction, with requirements, references, and verification cases fixed before evaluation.Repository cases must fail on the initial version and pass on the final reference; construction cases must fail on the starter and pass on the completed reference.Source: E BENCHMARK CURATION · p. 25, §E.1.1 · p. 26, §E.1.3 · p. 26

E.1.2 CROSS-CUTTING FEATURE EXTENSION

  • The controlled study compares Relic and structured controls across three models, ten workloads, and three seeds, while keeping shared tools, task-visible information, decision parameters, and execution code matched.Cross-cutting requirements expose dependent importer edits when public symbols or signatures change, and workload dependencies create recurring implementation, verification, and integration demands.Source: §E.1.2 · p. 26, §E.1.3 · p. 26

E.2.1 FROZEN SOURCES AND SPECIFICATIONS

The appendix specifies the frozen workloads, scoring assets, evaluator configurations, experimental conditions, and execution and validation procedures used to support reproducibility. It also defines a cross-run transfer test that isolates executable protocol bindings from identical readable rules while withholding source-run solutions and member-local state.

E.2.1 FROZEN SOURCES AND SPECIFICATIONS

  • W01–W10 map consistently across manifests, results, transfer records, and case studies, while frozen manifests link each workload to its source artifacts and qualification record.Packs are excluded for unstable or manual evaluation, incompatible licensing, already-satisfied starter behavior, solution leakage, irreproducible dependencies, or absent organizational demand.Source: §E.2.1 · p. 27, §E.2.3 · p. 27

E.2.2 INCLUSION CRITERIA

  • Retained public-gate coverage is 12/12 for W06, 8/12 for W07, 6/9 for W08, 7/12 for W09, and 9/12 for W10; checks are red on the starter and green on the reference.Usability requires a working environment and merge gate, a verifier whose retained scored cases fail on the untouched starter and pass on the fixed reference, and an agent-visible gate red on the untouched starter.Source: §E.2.2 · p. 27

E.3.1 STARTER ARTIFACTS

  • W01–W05 use newly authored program skeletons and references, whereas W06–W10 use frozen initial repositories and reserve their final versions as references.Each pack includes public requirements, held-out requirements, hidden tests, a reference implementation, and manifest-declared scoring units and evaluator entry points.Source: §E.3.1 · p. 28, §E.3.2 · p. 28

E.4.3 EXPOSED AND HELD-OUT BEHAVIORAL CASES

  • Exposed units give the organization a visible requirement and public acceptance check, while held-out requirements remain unseen and are scored offline on final mainline states and saved checkpoints.A complete contract requires every constituent behavioral case to pass; in W01, each of five contracts has seven cases, and qualification gives the starter 0/5 and reference 5/5 complete contracts.Source: §E.4.3 · p. 29, §E.4.4 · p. 29

E.5.3 LOWER AND UPPER BOUNDS

  • Every retained scored case fails on the untouched starter and passes on the fixed reference, producing 0% and 100% qualification scores, respectively.The sources, references, and scoring rules are frozen before rollout; evaluator, plan, and candidate identities are recorded to link each score to the evaluated state.Source: §E.5.3 · p. 30, §E.5.2 · p. 30

E.6.1 MAIN-STUDY MATRIX

  • The main study comprises 360 runs across ten workloads, three seeds, four conditions, and three models, with 120 runs contributed by each model.B0 is a single reflective agent; B1 is a persistent role-based team; B2 adds SDL, profile conditioning, and member learning without protocol institutionalization; B3 enables the protocol lifecycle on the B2 backbone.Source: §E.6.1 · p. 30, §F.1 · p. 31, §F.1.1 · p. 31

F.2.1 ROSTER SIZE AND FUNCTIONAL SPECIALTIES

  • B1–B3 share an eight-member roster and functional approval structure, while fresh members are reinitialized with role slots preserved and no hidden-suite, reference, or held-out knowledge.B0 aggregates the canonical roster into one member; B2 and B3 share functional prior assignments, and B3 protocols can be amended or retired.Source: §F.2.1 · p. 32, §F.2.2 · p. 32, §F.1.4 · p. 32

F.3 PROMPTS AND INFORMATION CONTROLS

  • B0 and B1 use LLM action selection, whereas B2, B3, and text/transfer variants use SDL; model calls remain for activities such as editing, reflection, and proposal evaluation.Editors receive the target file, current content, public evidence, and edit goal, with readable organizational rules supplied in a dedicated block for the specified treatments.Source: §F.3 · p. 33, §F.3.3 · p. 34

F.4.1 MODEL AND DECODING CONFIGURATION

  • The main study uses GPT-5.6 Terra, Claude Opus 4.6, and DeepSeek-V4-Flash, with 120 runs per model, a 336-step horizon, 24-step checkpoints, and no matched hard token, model-call, or primary-action ceiling.Randomness is namespaced from the run seed, with separate streams for policy sampling, availability, sandbox timing, and other controls.Source: §F.4.1 · p. 34, §F.4.4 · p. 35, §F.4.5 · p. 35

F.5.1 INITIALIZATION AND PREFLIGHT

  • Preflight freezes run-specific code and verifies public-suite red/green behavior; execution checkpoints state and token totals every 24 steps, then scores checkpoints offline and grades final mainline endpoints.Interrupted runs resume from the last checkpoint under the same run identity.Source: §F.5.1 · p. 35, §F.5.2 · p. 36, §F.5.3 · p. 36

F.6.1 MECHANISM-ISOLATION TESTS

  • The conditions share candidate-generation, decision, execution, registry, model, and member-information components, while tests verify allowed mechanism switches, delivery-action reachability, configuration matching, and member-information restrictions.The B2–B3 mechanism switch is institutionalization; access tests cover private evaluator assets, future repository history, private workspaces, and live-network access.Source: §F.6.1 · p. 37, §F.6.2 · p. 37, §F.6.3 · p. 37

G.1.1 SOURCE-RUN ELIGIBILITY

  • The transfer package derives from recurring B3 capability families, normalizes four source concerns into six typed runtime guards, and is selected without target-evaluation outcomes.Compatibility checks verify trigger observability and action reachability; Table 21 maps the concerns to guards.Source: §G.1.1 · p. 37, Table 21 · p. 38

G.2.1 FRESH ROSTER

  • Targets use fresh eight-role rosters and reset member-local state, while transferring only validated organizational rules, responsibility mappings, and referenced carrier documents—not source code, patches, hidden-test outcomes, or task answers.Text and Exec each use 30 target runs across ten workloads and three seeds; Fresh reuses the corresponding 30 B2 runs without an inherited package.Source: G INTERNAL PROTOCOL TRANSFER AND… · p. 37, §G.2.1 · p. 38, §G.4.1 · p. 39

G INTERNAL PROTOCOL TRANSFER AND EXECUTABLE BINDING

  • Text and Exec transfer the same frozen readable package, but only Exec activates six compiled runtime bindings; both conditions disable new protocol adoption and revision during the target window.The package was frozen before evaluation; ordinary code edits, tests, communication, local memory, and B2 member-learning paths continue.Source: §G.3.4 · p. 39, §G.5.1 · p. 40, §G.5.2 · p. 40

G.6 TRANSFER ESTIMANDS AND EVALUATION SCOPE

The section defines internal-transfer comparisons and explains how repository outcomes, evaluator results, protocol events, and resource use are recorded. These records connect reported measures to underlying work objects and saved run state.

G.6.1 PRIMARY CONTROLLED COMPARISONS

  • Table 23 presents the internal-transfer comparisons, with the supplied passage identifying the table’s scope but not its cell values.Source: Table 23 · p. 40

G.6.2 INTERPRETATION

  • Exec–Text estimates executable binding’s gain over a content-matched source-derived package, while Exec–Fresh estimates its gain over fresh B2 initialization.Source: §G.6.2 · p. 40

H.1.1 REPOSITORY STATE AND ACTION LOG

  • Repository and product outcomes are joined to evaluator outputs, while the action log records explicit action occurrences and retains object identifiers for tracing counts.Source: §H.1.1 · p. 40

H.1.2 EVALUATOR OUTPUTS

  • Evaluation artifacts record task, command, owner, status, repository digests, execution policy, exit code, and output hashes; passing requires candidate status passed.A confirmed fix additionally requires base status failed, and checkpoint evaluation uses the same scoring fields.Source: §H.1.2 · p. 41

H.1.3 ORGANIZATIONAL-STATE AND PROTOCOL RECORDS

  • Protocol adoption, use, violation, enforcement, amendment, and retirement are tracked as separate event types, with governed targets linked to protocol lineage.Per-spec counters cross-check ledger totals; enforcement references include PRs, patches, commits, and other work objects.Source: §H.1.3 · p. 41

H.1.4 TOKEN AND COMPUTE RECEIPTS

  • Provider-token usage, including retries, is recorded separately from elapsed run duration, and checkpoint token totals support matched-spend analysis.Tool and container compute follows a separate resource-accounting definition.Source: §H.1.4 · p. 41

H.2 METRIC DICTIONARY

This appendix defines outcome and process metrics, their populations, and aggregation procedures across the main study and protocol census. It also reports protocol adoption and formation counts in the Terra+Opus B3 census.

H.2.4 EFFICIENCY METRICS

  • Efficiency measures include tokens per task, tokens per confirmed fix, and actions per outcome, with eligible block ratios macro-averaged by arm.The paired B3–B2 cost contrast uses the same 20 common blocks.Source: Metric dictionary · p. 42, §H.2.4 · pp. 42–43

H.2.1 VERIFIED PRODUCTION METRICS

  • Verified production rates use frozen evaluator records, with complete contracts requiring every constituent behavioral case to pass on mainline.Exposed and held-out outcomes use their own requirement sets and scoring units; partial credit is retained at case level.Source: §H.2.1 · p. 42

H.2.2 WORKSPACE AND DELIVERY METRICS

  • Workspace metrics score local candidate trees, whereas mainline metrics score the integrated repository; delivery accounting follows generated, accepted, and merged patches and pull requests.Repository objects and explicit action occurrences are counted separately.Source: §H.2.2 · p. 42

H.2.3 ORGANIZATIONAL-MECHANISM METRICS

  • Protocol measures track proposal through retirement, retaining protocol identity and time for each event; use, decision effects, and execution effects have distinct records.The census groups revisions into parent lineages and measures persistence from a lifecycle origin to the last qualifying use or active step.Source: §H.2.3 · p. 42, §I.5.5 · p. 48

H.3.1 METRIC-SPECIFIC POOLING

  • Rates are averaged across seeds within model–workload–condition blocks, then equally weighted across applicable blocks; inapplicable or undefined outcomes remain NA.Measured zero outcomes retain zero, while infrastructure and malformed-evaluation errors retain explicit statuses.Source: §H.3.1 · p. 43, §H.3.3 · p. 43

H.4.1 PRIMARY ORGANIZATION CONTRAST

  • Primary organization contrasts pair B3 and B2 within model–workload–seed units, while transfer estimates and bootstrap intervals use their specified matched populations.The main study has 30 matched units per model; production endpoint intervals use 50,000 bootstrap draws, and the complete-contract contrast is primary.Source: §H.4.1 · p. 43, §H.4.3 · p. 44, §H.5.1 · p. 44

I.5.1 PROPOSALS AND ADOPTION

  • Across 60 Terra+Opus B3 runs, 617 terminal protocol objects included 513 adopted, 92 proposed, and 12 obsolete objects.The adopted objects comprised 393 autonomous and 120 environment-seeded rules.Source: Table 31 · p. 47, §I.5.1 · p. 47

I.5.5 CORPUS, COUNTING UNITS, AND FORMATION COVERAGE

  • 393 of 497 autonomous proposal lineages remained adopted at endpoint; 280 of 405 ever-adopted lineages were formed.The census classified 248 formed lineages as weak-only and 32 as strong.Source: Table 32 · p. 48, §I.5.5 · p. 48

I.5.6 SEMANTIC CLUSTERING AND FUNCTIONAL CATEGORIES

This section examines the reported protocols’ functional clusters—review, release, and evidence obligations—and their recurrence across workloads and controlled comparisons, alongside how recorded work episodes connect to protocol formation and shared delivery. It also traces adoption and delivery over time, including selective mechanism adoption and protocols amended and enforced through the endpoint.

I.5.6 SEMANTIC CLUSTERING AND FUNCTIONAL CATEGORIES

  • Review/merge accounts for 108 lineages, release engineering for 80, and evidence governance for 56, with distinct PR, release-candidate, and verification focuses.Review/merge and release protocols require current CI and evidence; release engineering additionally sequences ownership, patching, tests, sign-off, and rerunning the release gate.Source: §I.5.6 · p. 49

I.5.7 DIVERSITY, CONCENTRATION, AND CROSS-WORKLOAD REUSE

  • The three largest families comprise 244/280 lineages (87.1%), while review/merge, release engineering, and evidence governance recur across all 10 workloads.All 280 lineages show post-adoption use beyond their originating episode window; triggers, fields, and affected artifacts remain workload-specific within recurring families.Source: §I.5.7 · p. 50

I.6 MODEL-CONDITIONED FRICTION AND INSTITUTIONAL RESPONSES

  • Experiment activity is positively associated with experiment/evidence institutions (r = 0.24 [0.08, 0.44]), while feedback is negatively associated with customer/ownership institutions (r = −0.24 [−0.33, −0.14]).Debugging, claim-dispute, and launch-pressure correlations with corresponding institutions are small, with reported intervals spanning zero.Source: §I.6 · pp. 51–52

I.7.3 BUDGET-CAPPED CHECKPOINT SENSITIVITY

  • All four verified production contrasts remain positive at selected budget-capped B3 checkpoints.The sensitivity selects the last saved B3 checkpoint at or below B2's final token expenditure for each matched unit.Source: §I.7.3 · p. 52

I.8.2 PRIMARY CONTROLLED CONTRASTS

  • Exec improves behavioral-case pass rate by 6.5 pp over Text and 15.8 pp over Fresh, and exceeds Text on each production endpoint.The comparison is reported with 95% CIs of [0.7, 15.6] and [9.2, 31.8], respectively.Source: §I.8.2 · p. 53

I.9.2 MODEL-STRATIFIED B3–B2 EFFECTS

  • All four B3–B2 point estimates are positive in Terra, Opus 4.6, and DeepSeek, with no endpoint direction reversal across models.For DeepSeek held-out cases, B3–B2 is +2.78 pp [0.93, 5.56]; the three reported endpoints also remain positive under every workload omission.Source: Model-stratified B3–B2 effects · p. 54, Table 42 (continued) · p. 55

I.10.2 THREE-SEED OUTCOME COMPARISON

  • Executable B3 exceeds B3-text by 7.18 percentage points on complete contracts, with all four paired 95% intervals above zero.The same direction appears on held-out (+22.22 pp), exposed (+14.04 pp), and evaluator-confirmed seeded-issue (+11.71 pp) endpoints.Source: Table 43 · p. 55, Table 43 (continued) · p. 56

J.3.1 LOCAL PRODUCTION

  • In the Mini Blobstore case, B3 made 112 edit actions and had 118 accepted patches, while B1 made 169 edits and had 168 accepted patches.B1's last 96 steps contained no pull request opened or merged, whereas B3's final 96 steps included 26 patch commits and 18 proposal approvals.Source: §J.3.1 · p. 58, §J.3.2 · p. 58

J.3.3 DELIVERY OVER TIME

  • B3 adoptions occurred at t = 21, 39, 58, 60, and 65; all recorded baseline merges were in the first window, while B2 openings extended through t = 96–144.Adoption timing, PR creation, and merged delivery are shown as distinct series.Source: §J.3.3 · p. 58

J.4.1 ALL PROPOSED MECHANISMS

  • The run proposed ten mechanisms, adopted seven, and left three unadopted; Table 50 inventories the proposed, adopted, or unadopted mechanisms.A dash in the ledger denotes an unavailable count.Source: §J.4.2 · p. 59, Table 50 · p. 59, §J.4.1 · p. 59

J.4.2 ADOPTED AND REJECTED MECHANISMS

  • A merge-readiness handoff was adopted on its second attempt; P02 used a generic review-before-merge template, while P03 used a generic experiment-logging template.Source: §J.4.2 · p. 59

J.4.3 USES, ENFORCEMENTS, AND AMENDMENTS

  • 13 amendments across three protocols continued through t = 317; P01 recorded 211 enforcements across 12 pull requests, while P02–P06 had positive use but zero reported enforcements.At t = 336, the principal rule had been active for 315 steps and the current-commit integration-evidence rule for 276 steps.Source: §J.4.3 · p. 59

J.5.1 FIVE CONVERGENT RULES

The section describes mechanisms for making agent collaboration governable and reproducible: revisable rules bind review and readiness to verifiable evidence, while a liaison-first interface supports human participation through governance, translation, and attention management. It also covers the experiment’s isolation, recordkeeping, and recovery procedures.

J.5.1 FIVE CONVERGENT RULES

  • The five rules require contract-mapped change evidence, current-head verification, readiness checks, complete traces, and fresh mainline evidence before merge or release.CI and public smoke results must identify and pass on the current PR head; mainline movement requires reruns, while release claims are limited to verified dependencies.Source: §J.5.1 · pp. 59–60

J.6.2 REPEATED-VERIFICATION PATHOLOGY

  • B0 ran CI 73 times on one commit with no passes, while B1 logged 651 runs across 12 commits and passed two.B2 averaged 11.8 runs per commit and had five passes; B3 tested each of 93 commits about three and a half times.Source: §J.6.2 · p. 60, §J.5.3 · p. 60

J.7.1 ORIGINATING FRICTION AND CHALLENGE

  • After recurring contract-break events and a formal refusal demanding tests and evidence, M02 proposed the rule at step 12 and it was adopted at step 21.The rule entered the action context on adoption and was first used at step 22, producing a protocol-linked merge refusal at step 26.Source: §J.7.1 · p. 61, §J.7.2 · p. 61, §J.7.3 · p. 61

J.7.4 AMENDMENTS WHILE IN FORCE

  • The rule was amended twice while active, with recorded enforcement per step rising from 0.50 to 0.68 and from 0.55 to 0.69 across the two amendment intervals.M01 and M02 made the amendments; the proposal source for the second amendment was itself edited before adoption.Source: §J.7.4 · p. 61

J.7.5 LATER REUSE

  • The principal rule recorded 419 uses and 211 enforcements across 315 steps, affecting 12 pull requests and sustaining shared delivery.Eleven of the twelve requests later merged; the case interpretation traces recurring friction through adoption, enforcement, amendments, and delivery.Source: §J.7.5 · p. 61, §J.8.1 · p. 61, §J.9 · p. 62

J.8.2 DELAYED AND LATER-MERGED WORK

  • Eleven of 12 blocked requests later merged 15–112 steps after their first block, while one remained unmerged after 87 verifications and 65 of 76 merges were never blocked.The still-blocked request had reviewer-role approval but no observed override; the organization built a repair tool rather than overriding the gate.Source: §J.8.2 · p. 61, §J.8.3 · pp. 61–62, §J.8.4 · p. 62

K.1.1 FILESYSTEM AND PROCESS INTERFACES

  • The evaluation isolates hidden tests and reference material from members, scores an exported candidate workspace, and records hashes binding the score to that state.The evaluator can run in a separate network-disabled container or process, while members receive public-test and ordinary gate outputs.Source: §K.1.1 · p. 62, §K.1.2 · p. 62, §K.1.3 · p. 62

K.4.2 EXECUTION RECORDS

  • Versioned run records, execution logs, and hashed checkpoints preserve artifact identity and state for replay, validated resumption, and reconstruction of included outcomes.Resume checks scenario fingerprints before continuing; inclusion records retain completion, scoring eligibility, checkpoint lineage, and replacement reasons.Source: §K.4.2 · p. 63, §K.4.3 · p. 63, §K.5.1 · p. 64

K.7.1 EXACT CONDITION PROMPTS

  • The prompts turn recurring failures into checkable protocol proposals, distinguish protocol needs from repairs to existing rules, and leave adoption to governed review.Wish clustering requires recurring themes grounded in at least two wishes; proposal evaluation scores feasibility, usefulness, risk, and adoption but cannot approve or reject.Source: §K.7.1 · pp. 66–67

K.7.2 CONFIGURATION ARTIFACTS

  • Configuration artifacts specify workload inputs, schemas, prompts, transfer metadata, and target reset state; formative paired walkthroughs involved four author-team members who had not designed or implemented Relic or its interfaces.Walkthrough participants explored ongoing work, inspected blockers and evidence, communicated a requested change, and responded to decisions requiring human judgment.Source: §K.7.2 · p. 69, §L.6 · p. 73

L.1 WHY DIRECT HUMAN PARTICIPATION BECOMES DIFFICULT

  • Direct participation is difficult because humans must interpret structured organizational state and cannot assume the organization will pause or respond immediately to a natural-language message.The state includes ownership, reviews, CI, protocols, evidence, commitments, and memory; a secretary prototype reduced execution burden but left much of this complexity exposed.Source: §L.1 · pp. 69–70

L.2 MOVING THE HUMAN–ORGANIZATION ABSTRACTION BOUNDARY

  • The three interface levels progress from direct access, through a transparent secretary, to P3, which selectively surfaces organizational state while keeping detailed evidence available on demand.P1 places orientation and attention burdens on the human; P2 reduces some execution burden but retains substantial exposure to machine-native structures.Source: §L.2 · p. 70, Table 54 · p. 71
  • P3 makes the secretary the primary human-facing interface while the autonomous organization continues operating and human-owned artifacts remain subject to shared review, CI, and protocols.The human retains execution capacity through a human office, including delegated agents; organization-wide work remains governed by shared processes.Source: §L.2 · p. 70, §L.3 · p. 71

L.4 BIDIRECTIONAL TRANSLATION

  • The secretary translates organizational state into updates, decision requests, or on-demand explanations, and translates human intent into structured objects that follow ordinary validation, approval, and execution.This progressive disclosure keeps default summaries concise while preserving access to evidence and provenance.Source: §L.4 · pp. 71–72

L.5 HUMAN ATTENTION AS AN ORGANIZATIONAL RESOURCE

  • Human-attention policy must determine both which issues merit escalation and whether execution waits, slows, continues, or defers irreversible actions while judgment is pending.Routine, reversible issues may be handled internally; recurring non-urgent issues may be summarized periodically, while high-impact or unresolved decisions may be escalated.Source: §L.5 · pp. 72–73

L.7 PAIRED INTERNAL WALKTHROUGH RESULTS

The paired walkthroughs examine whether liaison-first summaries make organizational state easier to interpret and how CooperBench pair-level evaluation handles evaluator requirements unsupported by public specifications. They also report released-setting comparisons among plain peer cooperation, the structured lead–member Team, and Solo under a policy excluding 175 affected pairs linked to 33 shared contract issues.

L.7 PAIRED INTERNAL WALKTHROUGH RESULTS

  • P3’s liaison-first summaries reduced the burden of interpreting machine-native state, but human deliberation remained slower than agents’ ongoing work.The secretary surfaced issues requiring attention and let evaluators request underlying evidence or provenance; pausing could prevent state changes but made the organization wait for human decisions.Source: §L.7 · p. 73

M.1 FULL-BENCHMARK SCOPE AND SUCCESS CRITERION

  • CooperBench scores exact interacting-feature pairs: success requires the delivered candidate to pass both features’ official tests, and the ambiguity-aware validity-filtered set contains 477 pairs.Released-system results are recomputed on fixed 652-, 541-, and 477-pair universes; only an official PASS counts toward the numerator.Source: §M.1 · p. 74

M.2 TWO-MEMBER RELIC ADAPTATION

  • Relic’s two-member adaptation retains SDL, governed protocol formation, and executable bindings, with owner-local work integrated through review, replay, and shared-mainline synchronization.The benchmark evaluation uses Claude Opus 4.6 with high reasoning effort; members can propose, validate, and approve rules from visible coordination failures for subsequent work.Source: §M.2 · p. 75

M.3 BASELINE CONFIGURATIONS

  • Released comparisons include Solo, peer/coop+git, and hierarchical Team systems, whose reported coordination ordering is Peer < Solo < Team; the controlled 48-pair comparison holds the model constant across Relic, Official Peer, and Official Solo.Relic retains the peer topology without a permanent lead and adds persistent organizational state and executable governance.Source: §M.3 · p. 75

M.4.1 HIDDEN TESTS VERSUS HIDDEN REQUIREMENTS

  • The audit asks whether evaluator-enforced behavior is supported by the public contract, distinguishing legitimate hidden test instances from hidden requirements.For example, unseen valid JSON inputs may test a public JSON requirement, whereas requiring YAML when only JSON and CSV are specified adds an unpublished requirement.Source: §M.4.1 · p. 75

M.4.2 AUDIT SCOPE AND OUTCOME DISTRIBUTION

  • 33 specification–verification issue families affect 175 distinct exact feature pairs, and the audit excludes pairs only from aggregate scoring without rewriting their original outcomes.Eleven pairs instantiate two issue families, so family-level incidences sum to 186; the ambiguity-aware exclusion set contains 21 PASS, 118 FAIL, and 36 pairs without an official verdict.Source: §M.4.2 · p. 76

M.4.3 VALIDITY EVIDENCE LEVELS

  • The audit distinguishes 21 definite-defect families affecting 111 unique pairs from 12 ambiguity families affecting 71 pairs; the ambiguity-aware policy excludes all 33 families and 175 pairs.Seven pairs appear in both evidence levels, leaving 64 additional unique pairs contributed by ambiguity families.Source: §M.4.3 · p. 76

M.4.4 COMPLETE ISSUE-FAMILY INVENTORY

  • The issue-family inventory includes unpublished interfaces, underspecified behaviors, and cross-feature incompatibilities, such as evaluator-required YAML beyond a public JSON/CSV contract and retained tests tied to an older internal API.These examples illustrate distinct contract problems identified in the audit rather than benchmark outcome comparisons.Source: §M.4.4 · pp. 77–78
  • Pillow’s two feature contracts conflict: one requires preserving readonly state on save-as, while the other requires the default behavior to make the destination writable.The same default save-as operation cannot satisfy both public feature contracts.Source: §M.4.4 · p. 78
  • 175 excluded pairs arise from 33 shared feature- or pair-level contract issues propagated through CooperBench’s pairwise composition.Exact feature-pair scoring spreads defects across units: examples include click/2068/f10 affecting 11 pairs and tiktoken, jinja, and outlines issues affecting nine each.Source: §M.4.4 · pp. 78–79

M.5 RELEASED-BASELINE SENSITIVITY

  • Peer / coop+git < Solo < Team no-proto; this structural ordering holds across validity policies, although absolute pass rates change.In released CooperBench settings, splitting interacting features across peers reduces performance relative to Solo, while the structured lead–member Team exceeds Solo.Source: §M.5 · p. 79

M.6 RELIC RESULTS AND MATCHED-PAIR SENSITIVITY

Relic’s peer organization reaches hierarchical-Team-level performance on the raw matched CooperBench set and exceeds Team under stricter validity policies. On a fixed same-model subset, it also surpasses Solo while retaining the two-peer structure.

M.6 RELIC RESULTS AND MATCHED-PAIR SENSITIVITY

  • On 616 raw matched pairs, Relic reaches 63.0%, above Peer (51.5%) and Solo (56.8%) and 0.5 points below Team (63.5%).The raw comparison reports gains of +11.5 points over Peer and +6.2 over Solo.Source: §M.6 · p. 79, Table 58 · p. 79
  • Under the definite-defect policy, Relic scores 71.6% versus Peer’s 56.6%, Solo’s 63.0%, and Team’s 70.7% on the same 535 pairs.Its differences are +15.0 points over Peer, +8.6 over Solo, and +0.9 over Team.Source: §M.6 · pp. 79–80, Table 58 · p. 79
  • 76.9% on the 477-pair ambiguity-aware set puts Relic above Peer (59.1%), Solo (66.2%), and Team (74.2%).The differences are +17.8 points over Peer, +10.7 over Solo, and +2.7 over Team.Source: Table 58 · p. 79, §M.6 · p. 80

M.7 FROZEN 48-PAIR SAME-MODEL COORDINATION CHECK

  • On the fixed 48-pair same-model subset, Relic scores 29/48, exceeding Solo’s 26/48 while Official Peer falls to 13/48.The comparison holds model, reasoning setting, pair identities, and two-peer feature split constant, with no permanent lead.Source: §M.7 · p. 80, §M.7.1 · p. 81, Table 60 · p. 81

N EXTERNAL SOFTWARE BENCHMARK: PROGRAMBENCH

  • ProgramBench evaluates an executable protocol layer attached to the native Bash-action interface of a single mini-SWE-agent coding agent.The system configuration describes a stateful sidecar that checks proposed actions and accumulated execution evidence before dispatch.Source: N EXTERNAL SOFTWARE BENCHMARK:… · p. 81, §N.1 · p. 81

N.1 SYSTEM CONFIGURATION

  • The ProgramBench treatment keeps the model in control of actions, code edits, conversation history, and submission, while the sidecar allows, requests evidence for, or refuses transitions.The protocol package is frozen during target runs.Source: §N.1 · p. 81

N.2 TASK SELECTION AND FIXED EVALUATION SET

The fixed ProgramBench evaluation pairs a protocol treatment with official mini-SWE-agent. The treatment uses a frozen runtime-bound protocol package, records both blocking and advisory activity, and improves native-evaluator pass percentage on the sampled tasks.

N.2 TASK SELECTION AND FIXED EVALUATION SET

  • The evaluation randomly sampled 25 ProgramBench tasks, fixed their identities, and ran both systems on that paired set.The sample contains 17 Rust, three C++, three C, and two Go tasks.Source: §N.2 · pp. 81–82, Table 61 · p. 82

N.3 FROZEN PROTOCOL PACKAGE AND RUNTIME BINDING

  • The frozen package contains six executable binding rules and three advisory rules, mapped to the native interface’s triggers, evidence fields, and actions.Binding rules can affect execution; advisory rules are recorded as shadow guidance and do not themselves block an action.Source: §N.3 · p. 82, Table 62 · p. 82

N.4 RUNTIME FOOTPRINT

  • Across 2,215 Bash actions, the protocol layer records 933 validation receipts and 815 inspections, and issues 40 hard refusals.The receipts comprise 370 PASS, 48 FAIL, and 515 INCONCLUSIVE outcomes; 293 advisory shadow cases do not block execution.Source: §N.4 · p. 82

N.5 SCORING AND OUTCOMES

  • Behavioral pass percentage uses ProgramBench’s native evaluator, excludes its ignored tests, and is macro-averaged over the fixed 25 tasks.Both systems use the same scoring procedure.Source: §N.5 · p. 83
  • 70.916% versus 64.164% for official mini-SWE-agent yields a 6.752-point absolute gain on the fixed 25-task set.All 25 treatment trajectories produced submissions and completed evaluations.Source: §N.5 · p. 83
Loading 2609.32965v1…