OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime

1 Introduction

• Configured roles may not collaborate in execution, while harness differences and run-to-run variance can obscure what produced performance gaps.Existing evaluations often report final outcomes without verifying whether collaboration took place.Source: §1 · pp. 1–2
• OpenCollab declares role boundaries and communication topologies, holds non-organizational factors constant on a shared runtime, and records actions and resource use in fine-grained event streams.Adherence quantifies realized organization and supports per-run checks relevant to attributing the complier average causal effect (CACE).Source: §1 · pp. 2–3
• 64.25% on SWE-bench Pro, 83.15% on Terminal-Bench 2.1, and 69.91% on DeepSWE establish new SOTA against Claude Code, Codex CLI, and Mini-SWE-agent.The paper also reports Duo gains over Base of +3.4 points on Terminal-Bench 2.1 and +14.2 points on DeepSWE.Source: §1 · pp. 2–3
• 47.2% to 97.2%: changing any single configuration dimension shifts Adherence, while at 47.2% lead agents may monopolize budgets or bypass teammates.The five dimensions are model, tool set, budget, context, and topology.Source: §1 · p. 2

2 Motivation

• Declared organizations frequently fall below half Adherence as lead agents bypass teammates or exhaust their token allowance before handing off work.The paper calls this divergence between configured and executed organizations the illusion of collaboration.Source: §2 · p. 3
• Prompt-mandated delegation makes Adherence swing with phrasing, while filtering to collaborative runs selects harder tasks and biases comparisons against multi-agent systems.Agents may solve easy tasks alone and seek help only when stuck, so collaborative-run subsets skew toward harder tasks.Source: §2 · p. 3
• Isolated execution loops allow local budget exhaustion, and prose logs lack role-level attribution, obscuring tool use, costs, and whether topology edges were traversed.The proposed infrastructure combines shared-runtime pre-call admission, role-stamped event streams, and Adherence-based CACE analysis.Source: §2 · pp. 3–4

3.1 Architecture Overview

• OpenCollab decouples organization design from execution mechanics: independent controllers govern topology, lifecycle, and messaging over a shared session runtime.Sessions use an identical state-machine loop for context management, model calls, and tool execution, supporting unit consistency across runs.Source: §3.1 · p. 4

3.2 Programmable Collaboration

• 239 lines in OpenCollab reproduce the core multi-agent pattern of edict, reported as approximately 24,000 lines of code.Team mode lets a model choose whether and when to delegate within declared roles, tools, and communication edges; Workflow scripts specify call order, fan-out, and conditional transitions.Source: §3.2 · p. 5

3.3 Controlled Evaluation and Comparison

• Seven controlled-comparison conditions cover five runtime factors and two verification requirements; OpenCollab meets all seven, while none of ten audited artifacts does.The five factors include model, tools, token and cost budgets, context policy, and topology; verification records declarations and realized trajectories.Source: §3.3 · p. 5

3.4 Execution Traceability and Observability

• An ordered, append-only event stream records each step with run, role, event-type, and typed-payload fields, enabling reconstruction of execution topology and attribution of resource costs.Non-entry team roles use isolated Git worktrees, while only the entry-role workspace is graded.Source: §3.4 · p. 5

3.5 Evaluation of Cooperation Effectiveness

• A Team run is adherent only if every applicable verification axis passes; any deviant or unverifiable axis makes its Adherence indicator zero.The axes cover participation, delegation, role boundary, budget sharing, information flow, and context policy; delegation alone is insufficient when a required teammate does not act.Source: §3.5 · p. 6
• CACE divides the Pass@1 difference from Single by the Adherence rate, identifying an effect among fully realized Team runs only under one-sided compliance and mean exclusion.Recording the outcome contrast for non-adherent Team runs provides evidence about mean exclusion, not proof of it; the paper also defines cost metrics but does not report them in its experiments.Source: §3.5 · p. 6

4 Experiments

• The experiments evaluate SWE-bench Pro, Terminal-Bench 2.1, and DeepSWE, comparing OpenCollab with Claude Code, Codex CLI, and Mini-SWE-agent across three models.The models are DeepSeek-V4.1-Flash, Qwen3.8-Flash, and GPT-5.6-Luna.Source: §4 · p. 6

4.1 System-Level Performance and Organizational Value

• OpenCollab (Duo) attains the highest Pass@1 on SWE-bench Pro, Terminal-Bench 2.1, and DeepSWE, at 64.25%, 83.15%, and 69.91%, respectively.Table 2 covers cross-harness performance on those three benchmarks.Source: Table 2 · p. 7, §4.1 · p. 7
• Duo solves 21 DeepSWE tasks that Base fails and fails 5 that Base solves (69.91% vs. 55.75%); on Terminal-Bench 2.1 the split is 6 to 3 (83.15% vs. 79.78%).The Terminal-Bench task split is too small to separate the two; Duo spends more tokens by running two complete solving processes and selecting between candidates.Source: §4.1 · p. 7

4.2 Deconstructing Collaboration: Dimensions of Adherence

• Against a Qwen3.8-Flash reference team, unconstrained defaults yield 47.2% Adherence, while constraints such as restricted tool boundaries raise it above 90%.The experiment changes one configuration dimension at a time; Table 3 reports single-dimension ablations relative to the reference team.Source: Table 3 · p. 7, §4.2 · p. 8

4.3 Causal Validity and Trace Diagnostics

• With the Open card, 47.2% Adherence rescales an ITT of +5.6 to +11.8, but the authors caution that +11.8 is not the complier effect.The paired deployment contrasts are reported for configurations on SWE-bench Pro.Source: §4.3 · p. 8, Table 4 · p. 8
• Runtime traces show the reference adopter monopolizing execution and not submitting teammate work, while restricting it to read-only tools leads to delegation and adoption of Coder A’s patch.The restriction removes the adopter’s ability to edit files.Source: §4.3 · p. 8

• Modern frameworks use multi-agent interaction, stateful graphs, or shared-state management, but often entangle organizational design with low-level execution.OpenCollab separates orchestration from runtime to evaluate structures under identical system mechanics.Source: §5 · pp. 8–9
• Classical organization theory informs agent architectures, while contemporary frameworks operationalize them through interaction, stateful graphs, and shared-state management.These frameworks lower the engineering barrier to deploying complex collaborative topologies.Source: §5 · pp. 8–9
• Prior results on adding agents are mixed or negative, with coordination overhead and uncontrolled harnesses complicating comparisons (Zhang et al., 2026).The cited studies also report gains below replication noise floors and mixed returns under equal token budgets.Source: §5 · p. 9
• Existing frameworks often report outcomes without verifying actual communication, despite studies of coordinator roles, plan compliance, execution traces, and role separation.OpenCollab uses structured event streams and Adherence to audit delivery under explicit identification assumptions.Source: §5 · p. 9

6 Conclusion

• OpenCollab decouples organizational design from execution on a shared substrate, tracks runtime Adherence, and concludes that unverified collaboration prevents attribution of gains.It provides unified programmability and controlled runtime for evaluating organizational effects.Source: §6 · p. 9

A.1 Setup and notation

• A Team run is adherent only when every applicable axis is explicitly judged adherent; failed or unverifiable verdicts count as non-adherent.The common assumptions include consistency, isolated resets, stable model and runtime versions, and complete scoring; failures and timeouts are not dropped based on outcomes.Source: §A.1 · p. 13
• Single cannot realize the target Team organization, so the compliance pair is either (0, 0) or (0, 1), giving one-sided compliance.This is the monotonicity property relevant to the Wald argument, not a claim that Single fails to adhere to its own topology.Source: §A.1 · p. 13
• The configuration-level CACE is the effect of assigning the target Team configuration rather than Single among fully adherent runs, not an isolated effect of coordination.The non-adherent stratum includes more than non-delegating runs.Source: §A.1 · pp. 13–14

A.2 CACE decomposition and Wald identification

• The ITT/αadh Wald ratio identifies CACE only under mean exclusion and the common conditions; the ratio is not an unconditional identity.The derivation decomposes the outcome contrast by adherence status, and the Single arm’s inability to realize the target Team organization supports the complier interpretation, not the ratio by itself.Source: A Identification theory for the CACE · p. 13, §A.2 · p. 14

A.3 Paired diagnostic for the exclusion restriction

• Compatible task-paired contrasts estimate CACE among adherent runs and the non-adherent contrast relevant to mean exclusion; the latter is evidence about, not proof of, that restriction.The pairing requires a specified cross-arm coupling; sharing a task identifier alone does not establish it.Source: §A.3 · p. 14

A.4 Sensitivity to exclusion violations

• Without mean exclusion, the Wald-ratio discrepancy is bounded using the non-adherent contrast and adherence rate, but the resulting interval need not be sharp.A diagnostic limited to non-delegating runs does not bound the full non-adherent stratum unless it also covers the remaining A = 0 runs.Source: §A.4 · p. 15

A.5 Reporting requirements

• Reports should distinguish the observation regime and CACE estimation route, retaining task, arm, outcome, per-axis verdicts, and any pairing protocol.The table separates the deployment effect from two ways of learning about CACE; common scoring, consistency, reset, stability, and task-distribution conditions apply.Source: §A.5 · p. 15, Table 5 · p. 15

A.6 Empirical check on the adherence experiment

• The analysis pairs each configuration with Single on the same 36 tasks using product coupling conditional on task; failed and unverifiable Team runs have Ar = 0.Table 6 reports the adherent and non-adherent paired contrasts alongside the Wald ratio.Source: §A.6 · pp. 15–16, Table 6 · p. 16
• 17.6 points at 47.2%, 3.7 at 75.0%, and at most 2.9 above 90%: the Wald-to-paired estimate gap narrows as Adherence rises.Under the Mandatory card, the three non-adherent tasks show no difference and the two estimates coincide.Source: Table 6 · p. 16, §A.6 · p. 16
• On the Open card, the +5.6 ITT comes from 19 non-adherent tasks, while 17 adherent tasks show −5.9; the +11.8 Wald ratio should not be read as CACE.Among the 17 adherent tasks, the arms disagree on only one task, so the sign of −5.9 is not established.Source: §A.6 · p. 16
• These estimates are descriptive: rerunning unchanged Single flips 5 of 36 outcomes, and the table does not establish a collaboration effect for any configuration.The reported pattern supports the direction of the Wald–paired gap across Adherence levels, not a causal claim about an individual configuration.Source: §A.6 · p. 16

B Supplementary Experiments

• The supplement covers experiment setup, result tables, run endings and costs, traced runs, and limitations; Appendix A gives identification theory and Appendix C describes the system.Source: B Supplementary Experiments · p. 16

B.1 Experimental Setup

• The cross-harness comparison uses 193 SWE-bench Pro tasks, 89 Terminal-Bench 2.1 tasks, and 113 DeepSWE tasks; adherence uses the same 36 randomly drawn SWE-bench Pro tasks per configuration.Duo–single comparisons use the cross-harness task sets and are paired task by task on Terminal-Bench 2.1 and DeepSWE.Source: §B.1 · p. 16
• OpenCollab compares a single agent, a code-directed Duo Workflow, and model-directed Teams in which agents choose whether and how to message teammates.Duo has two Coders produce candidate fixes and code selects one; Star and Ring are the Team rosters used in the adherence experiment.Source: §B.1 · p. 17
• All configurations share a runtime and common execution controls, including step limits and run cutoffs; adherence runs have no network access inside agent containers.Agents have separate token allowances unless a Budget row changes them, and the runtime rejects calls that would exceed an allowance.Source: §B.1 · pp. 16–17
• Model-caused early endings remain in each denominator, while infrastructure faults trigger a rerun on the same task and code.A rerun still lacking a result counts as unverified for Adherence and unresolved for Pass@1.Source: §B.1 · p. 17
• Adherence requires every applicable axis to pass; for these Teams, every teammate must spend tokens and produce a model output, while Pass@1 is the resolved-run share.Unverified runs count as not adherent and unresolved; intervals are Clopper–Pearson 95%, and paired comparisons use an exact two-sided sign test without multiple-comparison correction.Source: §B.1 · p. 17

B.1.1 The adherence experiment

• Each adherence-experiment configuration runs once on the same 36 tasks; most rows vary one setting of the reference Star team, while two use Ring rosters.The unchanged single-agent configuration is run twice to measure run-to-run variation; model rows use DeepSeek-V4.1-Flash or GPT-5.6-Luna in every role.Source: §B.1.1 · p. 17, Table 7 · p. 18
• The experiment varies tools, budgets, card instructions, and communication topology, with the Optional-card row not strictly comparable to the other card rows.The card variants range from leaving choices open to prescribing work for Coders, and the Optional card permits the Adopter to implement the fix; a non-replying Coder could leave it waiting.Source: §B.1.1 · p. 18

B.1.2 The cross-harness comparison

• The cross-harness comparison runs five harnesses with GPT-5.6-Luna across three benchmarks; passes obtained with answer-leaking material were withdrawn or replaced.The four harnesses with observed successful external fetches were OpenCollab Base, OpenCollab Duo, Claude Code, and mini-swe-agent; no reference-material fetch was found for Codex CLI.Source: §B.1.2 · pp. 18–19

B.1.3 The Duo–single comparison

• Duo uses code to issue every handoff, runs two Coders on each task, and selects between their candidate fixes, calling a read-only Adjudicator when its fixed rule does not decide.Coder A targets a narrow root cause; Coder B targets completeness across components and boundary conditions.Source: §B.1.3 · pp. 19–20
• The paired Duo–Base difference is significant on DeepSWE but not on Terminal-Bench 2.1.Table 9 presents the task-paired comparison and exact two-sided sign test.Source: §B.1.3 · p. 20, Table 9 · p. 20

B.2 Full Results

• Table 10 reports task-by-task comparisons over 36 tasks: a:b counts tasks only this configuration succeeded on versus only the other did, with the exact two-sided sign test shown in parentheses.Source: §B.2 · p. 20

B.3 How Runs End and What They Cost

• 89% of T2 runs complete, while one-third of B3 runs reach the wall-clock limit; in B1, 92% stop for budget and 8% complete.Figure 5 summarizes completion, token-budget, wall-clock, and failed endings as shares of each configuration’s 36 runs.Source: §B.3 · pp. 20–21, Figure 5 · p. 21
• Single-agent runs average 1.3M–1.4M tokens, while team runs average 2.9M–4.2M except B1 at 0.9M and B3 at 4.7M.The entry agent accounts for 33%–50% of team tokens except T2 at 17%; average duration is 28–30 minutes for single-agent runs and 44–59 minutes for teams except B1 at 20 minutes.Source: §B.3 · p. 21
• Across configurations, the entry agent spends nearly all tokens early, teammates take over in the middle, and the entry agent’s share rises again in the final time slice.In B1 the entry agent uses all tokens in the first three slices; in T2 it uses almost none in the middle.Source: §B.3 · p. 21

B.4 Three Runs, Traced

• In the Read-only run, the adopter adopts Coder A’s fix and asks Coder B to review it; the graded patch comes from A, while B independently writes an equivalent fix.The adopter cannot edit in this variant, but can read files and adopt a commit.Source: §B.4 · pp. 21, 23
• In the reference-team run, the adopter briefs both Coders but keeps implementing, runs out of budget before submitting, and receives a coder’s commit only after stopping.The failed target tests concerned a constant name absent from the task statement, not the missed delegation.Source: §B.4 · p. 23
• In another Read-only run, the adopter ships Coder B’s commit after Coder A recommends it, but B’s code fails a target permission-error test.Coder A’s recommendation followed probes that did not expose the permission-related failure.Source: §B.4 · p. 23
• The figures trace individual event streams and show organizational behaviors, but they do not compare the teams with a single agent or estimate their frequency.Each example is one selected run, and their time axes differ.Source: §B.4 · pp. 21, 23

B.5 Limitations

• The evidence covers three model snapshots, serial agent execution, and one Duo decomposition, so it does not estimate general model variation, true parallelism, or other workflows.A different script could be equally faithful and perform differently.Source: §B.5 · p. 23

C System Details

• The system-details appendix describes the runtime, its controllers, code organization, collaboration declarations, artifact audit, run record, and adherence computation.Source: C System Details · p. 25

C.1 Runtime and Control Regimes

• Team, Workflow, and single-agent runs share a session runtime; controllers determine which sessions start and how work passes between them, while limits are checked before calls.Team models choose handoffs; Workflow scripts fix their order.Source: §C.1 · pp. 25–26

C.2 How the Code Is Organized

• The runtime separates domain, application, adapters, and bootstrap layers, checks their import contracts, and exposes a public surface through which the evaluator accesses it.Source: §C.2 · pp. 27–28

C.3 Declaring a Collaboration

• A team file specifies roles, tools, allowances, context policy, and directed message edges; the runtime refuses out-of-bound tool calls and undeclared messages.Every team needs a path back to its entry role so delegated work can be returned.Source: §C.3 · pp. 28–29

C.4 Audit of Ten Agent Artifacts

• The audit scores ten agent artifacts on seven conditions for setting and checking organizations; OpenCollab’s Team row sets roles and records departures, while other artifacts fall short in different ways.The audit uses pinned code versions for nine artifacts and vendor documentation for closed-source Claude Code.Source: §C.4 · pp. 29–30

C.5 The Run Record

• Each configuration produces a manifest, ordered event log, and metrics file joined by run ID, enabling role-level activity and costs to be read from events rather than prose logs.Events record timestamps, step numbers, run IDs, types, and role-named payloads.Source: §C.5 · pp. 30–31, Table 13 · p. 31
• Organizational overhead measures extra Team cost among non-adherent runs by pairing each task with its Single run, avoiding task-composition differences.Source: §C.5 · pp. 31–32
• Workload balance is normalized Shannon entropy over active-role token shares in fully adherent Team runs; values near one indicate more egalitarian effort.Source: §C.5 · p. 32

C.6 Adherence, Axis by Axis

• A run is adherent only if every applicable axis passes; in these Team experiments, each declared non-entry role must spend tokens and produce a model output.Unverifiable verdicts count against adherence, so missing evidence can lower but not raise the rate.Source: §C.6 · p. 32