PaperlayerView exact revision on arXiv

Source-linked AI summary

GitSwarm: Decentralized Compounding Inference

Vedant Shah, Ankur Samanta, Paras Dahal, Mikhail Plekhanov, Carole-Jean Wu, Scott Yih, Remi Munos, Rob Fergus, Jakob Foerster, Ruslan Salakhutdinov, Sanjeev Arora, Jason Weston, Aaron Courville, Anirudh Goyal

arXiv:2610.04862v1cs.AIcs.LG

TL;DR

Long-horizon problem solving and research require computation to accumulate across attempts, while much inference-time computation remains organized around individual trajectories or candidates. GitSwarm lets asynchronous workers preserve and extend work in a shared, branchable repository with explicit dependencies across branches. Across problem solving and GPU-backed research, the paper reports extensive reuse and improvements; GitSwarm's ProgramBench mean score is 79.39% versus 65.12% for Forced Codex.

  • Problem

    Long-horizon problem solving and research need computation to accumulate across attempts, but most inference-time computation is organized around individual trajectories or candidates rather than persistent reusable work.

    Source: §1 · p. 1

  • Method

    GitSwarm uses asynchronous workers and a shared branchable repository to preserve intermediate contributions and record explicit dependencies across branches.

    Source: §1 · p. 2

  • Results

    Across problem solving and GPU-backed research, GitSwarm reports extensive reuse and improvement over research starting points; its ProgramBench mean score is 79.39% versus 65.12% for Forced Codex.

    Source: §1 · pp. 2–3, §5 · p. 12

  • Takeaways & Limitations

    The substrate supports a continuing research loop in which agents preserve findings and use earlier evidence to guide new work.

    Source: §5 · p. 12

    Results are conditional on evaluated tasks and budgets; ProgramBench uses a fixed 50-task subset of 200, and the perfect IMOProofBench-Advanced result is a single run.

    Source: §4 · p. 12

Abstract

from arXiv · show

Long-horizon problem solving and scientific research require computation to accumulate across successive attempts. Partial solutions, experimental findings, and unsuccessful approaches can inform later work, yet most inference-time computation is organized around individual trajectories or candidates rather than a persistent body of reusable work. We call this paradigm compounding inference: organizing inference-time computation so that intermediate work persists and can be inspected, extended, combined, or challenged by subsequent computation. We instantiate compounding inference in GitSwarm, an asynchronous system where homogeneous agents independently decide how to advance a task while collaborating through structured persistent memory. Agents explore, experiment, verify, refine, and synthesize previous work in a shared, branch-able Git repository. Atomic commits preserve intermediate artifacts, while explicit semantic dependencies record how later contributions build on work across branches. We evaluate GitSwarm on long-horizon problem solving and sustained GPU-backed experimental research. On IMOProofBench-Advanced, GitSwarm solves all 30 problems in one run using GPT-5.5. On ProgramBench, it achieves a $79.4\%$ mean score, versus $65.1\%$ for the strongest reported baseline under the stated budget. On three neural architecture research tasks (Residual Matrix Transformer, Looped Transformer, NanoChat), GitSwarm improves upon the starting architectures through successive experimentation. Beyond final performance, we measure whether computation accumulates: on ProgramBench, $94.7\%$ of contributions are subsequently built upon, while the selected solution's ancestry covers $82-93\%$ of the contribution graph. These results show that inference-time computation can accumulate across otherwise independent episodes, forming an evolving body of work that subsequent inference can reuse.

1 Introduction

The introduction frames long-horizon problem solving and research as requiring computation to accumulate across attempts, then presents GitSwarm as a persistent, branchable approach evaluated on problem solving and research.

1 Introduction

  • Long-horizon work needs computation to accumulate across attempts; compounding inference organizes intermediate work as persistent, evolving, reusable work.Partial solutions, experimental findings, and unsuccessful approaches may inform later work.Source: §1 · p. 1
  • GitSwarm solves all 30 IMOProofBench-Advanced problems in one run and scores 79.39% on ProgramBench versus 65.12% for Forced Codex at comparable compute.Both results use Codex (GPT-5.5-High) workers.Source: §1 · p. 2
  • Across Residual Matrix Transformer, Looped Transformer, and NanoChat, GitSwarm improves starting points; Residual Matrix Transformer held-out losses are 2.890 and 2.894 versus 3.147 using 1.3B tokens.The compared configurations use the same 1.3B-token training budget as the reference.Source: §1 · p. 3
  • On ProgramBench, 94.7% of contributions are subsequently referenced, and selected-solution ancestry covers 81.6–92.6% of the contribution graph on average across budgets.Execution traces separately show artifact consumption; proof workers also reuse work to verify and compare candidates outside the selected solution's ancestry.Source: §1 · p. 3
  • The paper formulates compounding inference, instantiates it as branchable contributions with explicit cross-branch dependencies, and evaluates performance and accumulation across problem solving and research.Independent workers preserve competing lines of work and decide what to extend without a central intellectual planner.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 GitSwarm

GitSwarm organizes persistent work as a branchable computation graph, with workers deciding how to advance tasks and a harness coordinating execution and validation.

2 GitSwarm

  • GitSwarm uses asynchronous, homogeneous workers collaborating through a shared Git repository that preserves intermediate work as a branchable computation graph.The harness manages concurrency, isolation, validation, and publication.Source: §2 · p. 3, §2.5 · p. 4

2.1 Design Principles

  • Persistence, branchability, and attribution let discoveries survive their authors, competing approaches develop independently, and workers choose what to explore, verify, refine, or combine.The harness coordinates execution but does not assign intellectual roles or prescribe a work sequence.Source: §2.1 · p. 3

2.2 Shared Computation Graph

  • Each contribution is an atomic commit with a physical parent, while declared semantic sources add dependencies that let workers use discoveries from other branches without merging them.Semantic dependencies are worker-declared rather than independently established as causal influence.Source: §2.2 · pp. 3–4

2.3 Asynchronous Worker Protocol

  • Workers operate asynchronously in isolated worktrees and may contribute a validated atomic commit, nominate an existing candidate, or abstain when no useful work is identified.Validation enforces publication requirements, not solution correctness; voluntary abstentions trigger replacement without consuming the episode budget.Source: §2.3 · p. 4

2.4 Live Swarm State: Dashboard and In-Flight Activity

  • A dashboard reports run state without assigning work, while public in-flight logs expose workers' scopes and progress so peers can avoid duplicating active work.The dashboard adapts fields to task type, including shared compute usage and validation score for research tasks.Source: §2.4 · p. 4, Figure 2 · p. 5

2.5 Decentralized Readout

  • After the worker budget is exhausted, the harness selects the candidate with the most nominations, using deterministic tie-breaking while keeping nominations separate from the computation graph.Workers can evaluate existing candidates while others continue constructing new ones.Source: §2.5 · p. 4

3 Experiments

The experiments evaluate GitSwarm on fixed-answer proof and program tasks and GPU-backed architecture research, then measure reuse and adaptive coordination. They report benchmark gains, architecture improvements, and reuse of prior work across these settings.

3.1 Experimental Setup

  • The evaluation covers 30 mathematical proof problems and 50 ProgramBench reconstruction tasks, with worker-episode budgets of 20–160 and 100–500, respectively.The problem-solving comparison uses five concurrent GitSwarm workers and repeatedly continued single-agent baselines.Source: §3.1 · p. 5

3.2 Benchmark Performance and Inference-Time Scaling

  • GitSwarm reaches 79.4% mean score on ProgramBench versus 63.1–65.1% for the continued Codex baseline; GPT-5.5-high workers solve all 30 IMOProofBench-Advanced problems at budgets of 40 and 80.On IMOProofBench-Advanced, Best-of-N baselines also reach the benchmark ceiling at lower token consumption, so saturation alone does not establish a unique advantage for GitSwarm.Source: §3.2 · p. 6

3.3 Sustained Experimental Research

  • RMT swarms reach held-out losses of 2.890 and 2.894 versus the 3.147 reference at 1.3B tokens; Huginn reaches 3.187 versus the 3.265 reference after 26 hours.RMT’s final architecture includes a temporal mixer developed from an initially failed memory idea; Huginn’s adapter was rejected on 128 examples and later confirmed on 512.Source: §3.3 · p. 7
  • On NanoChat, 20 workers lower validation BPB from 0.993 to 0.905 in 1,465 successful evaluations; 70.1% of negative-labelled commits are later cited by accepted contributions.The incumbent combines changes developed on separate branches, and its capacity grows to approximately 1.72B parameters.Source: §3.3 · p. 8

3.4 Evidence for Compounding Computation

  • Later workers build on 94.7% of ProgramBench contributions and 44.4% of proof contributions; selected ProgramBench solutions reach 81.6–92.6% of the contribution graph on average.Proof ancestry covers 21.3–33.0% of the graph; later workers often compare existing proof candidates rather than extend the selected proof.Source: §3.4 · pp. 8–9
  • Execution traces show active reuse of 79.9–82.5% of eligible voluntary ProgramBench artifact bundles; these measurements establish reuse, not the causal contribution of individual dependencies to performance.On IMOProofBench-Advanced, workers open 583 of 759 computational probe bundles, and later workers execute or adapt 172 while citing their producers.Source: §3.4 · pp. 9–10

3.5 Adaptive Coordination and Design Tradeoffs

  • Without assigned roles, ProgramBench workers sustain implementation work late into runs, while proof workers shift early toward selecting existing proofs as budgets grow.Selected candidates first appear at 93.1–98.0% of ProgramBench run progress, compared with 8.6–17.5% for selected proofs; programming workers also inspect five to seven candidate commits, versus about three for proof workers.Source: §3.5 · pp. 10–11
  • On nine development tasks, asynchronous execution averages 38.0 minutes versus 49.3 for synchronized waves; decentralized nomination scores 62.06 ± 2.44% versus 64.40 ± 1.49% for central selection.These small experiments characterize latency and readout tradeoffs; they do not establish statistical equivalence or isolate persistent memory’s benefits.Source: §3.5 · p. 11

4 Discussion and Limitations

GitSwarm exhibits several forms of accumulated work, but the analyses do not isolate how much that reuse causes performance gains. The evaluation is bounded by task and budget choices, and persistent collaboration carries computational overhead.

4 Discussion and Limitations

  • Workers integrate discoveries across branches in program reconstruction; proof workers often verify or compare candidates without extending the selected proof’s lineage.Architecture experiments show cross-branch synthesis in RMT and recovery of an initially rejected idea in the Looped Transformer run.Source: §4 · p. 11
  • Dependency and execution-trace analyses show workers inspect and reuse earlier work, but do not isolate the resulting performance gains.Worker-declared informed_by dependencies and artifact consumption indicate use, not causal influence; selected-solution ancestry can omit useful verification and comparison.Source: §4 · p. 11
  • Synthetic experiments demonstrate operational benefits but do not establish the individual benchmark-performance contributions of persistence, branchability, or cross-branch dependencies.The proposed intervention resumes matched runs from the same candidate, with and without preceding research history, under equal remaining compute.Source: §4 · p. 12
  • Evaluation results are conditional on the tasks and budgets: ProgramBench uses 50 of 200 tasks, and the perfect proof result is a single run.Some baselines also reach the proof benchmark ceiling; comparisons vary in execution conditions, training tokens, inherited research, run status, starting checkpoint, and GPU allocation.Source: §4 · p. 12
  • Repository growth adds inspection-token costs, redundant or unsuccessful episodes can consume compute without accepted work, and GPU research adds contention and scheduling overhead.Whether accumulated-work benefits outweigh these costs requires longer-horizon evaluations with matched total compute and wall-clock budgets.Source: §4 · p. 12

5 Conclusion

The paper frames compounding inference as intermediate work persisting across independent agent episodes and presents GitSwarm as a substrate for this process. Its experiments show reuse across problem solving and GPU-backed research, while proposing reinforcement learning across diverse tasks as a next step.

5 Conclusion

  • Compounding inference organizes intermediate work so it persists across independent agent episodes.GitSwarm provides asynchronous workers, a shared branchable repository, and explicit cross-branch dependencies without a central intellectual planner.Source: §5 · p. 12
  • Across problem solving and GPU-backed research, agents reuse earlier work, synthesize independent discoveries, and recover initially unsuccessful ideas.The paper presents the substrate as a foundation for long-horizon discovery through a continuing cycle of proposals, experiments, preserved findings, and evidence-guided research.Source: §5 · p. 12
  • The authors propose reinforcement learning across diverse tasks in this environment to reward contributions whose value emerges through subsequent work.The proposed training objective would reward contributions beyond the outcome of an individual episode.Source: §5 · p. 12

AI Usage Statement

The work studies LLM-based digital agents accessed through provider APIs and uses LLMs for measurement and synthetic-task design. It also reports specific uses and non-uses of generative AI tools in implementation and research.

AI Usage Statement

  • Frontier models accessed through provider APIs served as the digital agents, while an LLM analyzer supported semantic judgments and LLMs helped design synthetic tasks.The passage identifies these as uses of LLMs in the work.Source: AI Usage Statement · p. 12
  • LLM coding agents assisted code implementation, which one author reviewed, and generated draft scientific figures.The authors state they reviewed all AI-assisted work and take responsibility for the final content and artifacts.Source: AI Usage Statement · p. 12
  • LLMs and generative AI tools were not used to develop theoretical models, formulate or prove mathematical claims, refine hypotheses, or design experiments.The passage also lists non-use in qualitative analysis, translation, dataset cleaning, result interpretation, and proof writing.Source: AI Usage Statement · p. 12

Ethics Statement

The work reports no human subjects, crowdsourced annotation, or personal data, and describes isolation and evaluation safeguards. It also identifies dual-use concerns and recommends human supervision and independent verification.

Ethics Statement

  • The experiments used public benchmarks and research tasks based on public papers, with workers isolated in Git worktrees and environments without internet access.Candidates were scored using fixed evaluators on protected development and separate held-out sets.Source: Ethics Statement · pp. 12–13
  • Program reconstruction could facilitate unauthorized re-implementation of proprietary software, while automated architecture research could accelerate AI development, including potentially misusable systems.The authors identify both capabilities as dual-use.Source: Ethics Statement · p. 13
  • The authors view GitSwarm as a human-supervised research tool and say intermediate conclusions should be independently verified before reliance.They note that small proxy evaluations mis-ranked candidate models and call for extensive behavioral analysis before deployment.Source: Ethics Statement · p. 13

A Related Work

GitSwarm is situated among systems for inference-time search, memory, coordination, and autonomous research, but differs by preserving a durable, branchable contribution history with explicit dependencies. The appendix outlines benchmark and trace-analysis details.

A Related Work

  • Related systems preserve intermediate work through within-search states, external memory, distilled summaries, or coordinated agent workflows; GitSwarm instead retains individual artifacts and their cross-branch dependencies.The cited comparisons include inference-time search, memory systems, multi-agent coordination, and autonomous research methods.Source: A Related Work · p. 18
  • GitSwarm separates each commit’s physical Git parent from mandatory semantic dependencies and selects final candidates through worker nominations and deterministic plurality readout.The comparison with Agora is complementary, not a direct performance comparison.Source: A Related Work · pp. 19–21
  • GitSwarm preserves a durable, branchable history of contributions and explicit dependencies, while workers independently choose what previous work to extend.Unlike shared-workspace systems focused on transient coordination, it records dependencies across branches and preserves competing approaches for later recombination.Source: A Related Work · pp. 18–19

B Experimental Setup Details

  • The appendix identifies its coverage: benchmark and model details, ProgramBench subsets, baseline continuation, and the run corpus for trace analyses.The systems comparison is by design principles, not performance ranking, and uses protocols described in cited papers.Source: A Related Work · p. 20, B Experimental Setup Details · p. 21

B.1 Benchmarks, Scoring, and Models

The appendix specifies benchmark scoring and model comparisons, task subsets, baseline continuation, and trace-analysis runs. It also introduces controlled synthetic tests of persistence, branchability, and attribution.

B.1 Benchmarks, Scoring, and Models

  • IMOProofBench-Advanced evaluates all 30 problems with proof scores of 0, 1, 6, or 7, counting a proof as fully correct at 7.The compared proof configurations include GPT-5.5-high GitSwarm and Forced Codex, plus Gemini-3.1-Pro GitSwarm and its mini-SWE-agent baseline.Source: §B.1 · p. 21
  • ProgramBench scores the fraction of hidden cases matched by a self-contained executable; GitSwarm Codex uses GPT-5.5 at high reasoning effort.The 50-task evaluation subset is stratified across implementation language, difficulty, test-suite size, and runtime, after excluding development and other tasks.Source: §B.1 · pp. 21–22, §B.2 · p. 22

B.2 ProgramBench Task Subsets

  • The nine ProgramBench development tasks are excluded from the evaluation subset, which contains 50 deterministically selected tasks.The development set is used for development and design-tradeoff comparisons.Source: §B.2 · p. 22

B.3 Baseline Continuation Protocol

  • Forced baselines are continued after each natural stop by prompting them to keep working, with intermediate checkpoints graded for compute-scaling curves.Natural stopping is the budget knob for this continuation protocol.Source: §B.3 · p. 22

B.4 Run Corpus for the Trace Analyses

  • Trace analyses use Gemini-3.1-Pro proof runs and ProgramBench runs at four budgets, B = 100–500, on the 50-task evaluation subset.Figure 11 illustrates how one worker’s observation can let another continue a Hidden Rulebook chain rather than rediscover its prefix.Source: §B.4 · p. 22, Figure 11 · p. 23

C Controlled Tests of Substrate Properties

  • The controlled synthetic tasks isolate persistence, branchability, and attribution as substrate properties, using Gemini-3.1-Pro and three random seeds.The appendix notes that symbols such as B and C are local to these tasks.Source: C Controlled Tests of Substrate… · p. 23

C.1 Persistence: Hidden Rulebook

In controlled synthetic tasks, shared persistence, branch-native memory, and explicit attribution reduce distinct forms of repeated work. The tests illustrate these properties’ potential usefulness but do not establish that they cause benchmark improvements.

C.1 Persistence: Hidden Rulebook

  • The Hidden Rulebook requires discovering opaque-ID fact and rule chains one record at a time; persistent workers can read prior workers’ completed observations.Relevant-record coverage counts distinct queried records belonging to requested chains, not correctness of final submissions.Source: §C.1 · pp. 23–24
  • Shared persistent memory maintains high relevant-record coverage as target count or chain depth grows, while independent baselines remain lower despite larger query budgets.As targets grow, coverage is about 70–75% for B = 2 and 20–23% for k = 4; the baselines also decline with depth.Source: §C.1 · p. 24, Figure 12 · p. 25

C.2 Branchability: Hidden Maze

  • Branch-native memory substantially reduces repeated traversal versus flat persistent memory across tested maze depths and branching factors.Restoring a reached endpoint lets a later worker explore another branch without re-traversing its known prefix.Source: Figure 13 · p. 25, Figure 14 · p. 26, §C.2 · p. 26
  • In the Hidden Maze, branch-native memory organizes completed trajectories beneath the endpoints they resumed and prompts workers to restore checkpoints for further exploration.The comparison is between practical memory interfaces: branch-native memory also makes checkpoint restoration explicit in the worker prompt.Source: §C.2 · pp. 26–27

C.3 Attribution: Retracted Evidence

  • In Retracted Evidence, attribution supplies producer and parent IDs; both treatments retain prior observations, but only one provides dependency metadata.The primary metric is the fraction of unique archive artifacts read before the first exact submission.Source: §C.3 · p. 27
  • Explicit attribution reduces the archive fraction workers must inspect across tested lineage depths, semantic ambiguity, and fan-out, with the gap widening as lineage deepens.Recorded dependencies let workers follow and invalidate affected lineages without reconstructing every dependency from artifact contents.Source: §C.3 · pp. 27–28, Figure 16 · p. 28

C.4 Scope of the Controlled Tests

  • The controlled tests do not establish that any individual substrate property causes GitSwarm’s benchmark improvements.That would require interventions within the full system while controlling for compute, worker behavior, and other substrate capabilities.Source: §C.4 · p. 28

D Benchmark Performance Details

Proof scaling curves show earlier saturation and more run-to-run variation than program reconstruction, while the configurations make cross-budget proof trends descriptive rather than controlled estimates of worker-count effects. The appendix also details the shared architecture-research setup and task-specific reporting.

D Benchmark Performance Details

  • Proof construction shows more run-to-run variation and earlier saturation than program reconstruction as budgets increase.The plotted proof configurations differ, so their cross-budget curves are descriptive rather than a controlled estimate of worker-count effects.Source: D Benchmark Performance Details · p. 28, Figure 17 · p. 29

E AutoResearch: Additional Details

  • The appendix provides task specifications, compared configurations, full-size hill-climbing figures, research records, and NanoChat comparison caveats.It also reports observations across the three architecture tasks.Source: E AutoResearch: Additional Details · p. 30

E.1 Common Setup

  • Across three architecture-improvement tasks, workers independently propose, implement, and test architectural modifications, using prior results to guide subsequent work.Workers share a fixed GPU allocation and submit jobs through SLURM; task outcomes are reported separately because objectives, budgets, and comparisons differ.Source: §E.1 · p. 30

E.2.1 Task Details

The RMT replaces the transformer residual vector with an outer-product memory matrix. The task asks the swarm to improve this architecture under a fixed parameter cap, training stream, and evaluation protocol.

E.2.1 Task Details

  • RMT replaces the residual vector with an outer-product memory matrix that layers read from and write to using learned keys and values.This lets the residual stream scale independently of parameters and compute.Source: §E.2.1 · p. 30
  • The task requires an improved RMT with a matrix-valued residual stream, at most 350M parameters, and training from scratch on 1.3B tokens.The search metric is next-token cross-entropy on a protected 4,873-example development set; the reference has development loss 3.158 and separate held-out loss 3.147.Source: §E.2.1 · p. 30
  • Direct GPT and Codex baselines use GPT-5.6-sol at xHigh reasoning for 72 hours on one eight-H200 node; the GitSwarm runs use 4 or 8 GPUs.The seeded GitSwarm run uses earlier incumbents, and Figure 4(a) shows the research trajectories.Source: §E.2.1 · p. 30

E.2.2 Final Architectures and Idea Families

GitSwarm improves the RMT reference within its original parameter cap and training-token budget through architectural changes, including mechanisms developed on separate branches. The research history also records how ideas were tested, reused, and combined across branches.

E.2.2 Final Architectures and Idea Families

  • The 8-GPU submission achieves held-out loss 2.890, while the 4-GPU run reaches 2.894; both improve on the reference within its parameter cap and training budget.Source: §E.2.2 · p. 31
  • GitSwarm’s gains come entirely from architectural changes at the reference training budget, while Codex’s 2.690 result came mainly from training on 10.1B tokens.Half of GitSwarm’s changes are specific to RMT’s matrix memory; about three quarters of Codex’s gain came from its longer training.Source: §E.2.2 · p. 31
  • Across 19 explored idea families, six entered the final model, seven were refuted, and 69 of 120 commits were published as negative results.About 30% of negative commits were later cited as evidence, compared with 84% of positive commits.Source: §E.2.2 · p. 31, Table 7 · p. 32

E.2.3 Reuse and Cross-Lineage Inheritance

  • The first durable improvement became an ancestor of 102 of 120 commits, and a failed single-token memory idea was later reused to build a successful temporal mixer.Four of the final model’s six mechanisms originated on side branches; evidence from other branches also contributed to the 24-commit, 24-worker dependency set.Source: §E.2.3 · pp. 31, 33, Table 8 · p. 33

E.2.4 Solution Path and Cross-Pollination Run

  • On the best-model lineage, commits added a depth fix (0.6 h), tied embeddings (1.8 h), SwiGLU (5.9 h), and a head router (34.6 h, loss 2.934).Source: §E.2.4 · p. 33

E.2.5 Findings about the Research Process •

  • Small-scale screening helped late less often: 15 of 17 promoted ideas helped during the 72-hour run, compared with 1 of 8 in the extension.A seed replication shifted loss by 0.010, comparable to several later solution-path steps; gains measured in isolation also sometimes vanished in combination.Source: §E.2.5 · p. 34

E.3.1 Task Details

  • The Looped Transformer task requires preserving weight-tied recurrence below 350M parameters and compares candidates at exactly eight recurrences.The reference is about 308M parameters and has loss 3.265; at the time of writing, the 12-worker, 48-hour runs were 26 hours in progress and used smaller-model screening before full-size training.Source: §E.3.1 · p. 34

E.3.2 Final Architecture and Idea Families

The Looped Transformer’s best model replaces its recurrent input adapter with a gated feed-forward network, while NanoChat shows improvement through joint search over architecture, capacity, optimization, and training systems. The supplied research histories also document reuse and measurement limits, without isolating the causal contribution of shared memory to performance.

E.3.2 Final Architecture and Idea Families

  • The final architecture replaces the linear adapter that re-injects input at every iteration with a small gated feed-forward network, adding 27M parameters (8.7%).Neither baseline had yet beaten the reference.Source: §E.3.2 · p. 34

E.3.3 Steps on the Solution Path

  • The adapter won at every model-size rung and reached 3.187 at full size after 9 hours, after an initial 128-example rejection was overturned by a 512-example re-check.The final model’s code came almost entirely from a six-commit lineage, while declared evidence dependencies span 89 commits by 89 workers.Source: §E.3.3 · pp. 35–36

E.3.4 Findings about the Research Process

  • In the Looped Transformer run, small evaluations disagreed with full-set rankings on 5 of 12 comparable pairs, prompting a 128→512→4,873 evaluation funnel.Workers also ran parameter-matched controls; after the evaluation budget ran out, most commits were calibrations and diagnostics.Source: §E.3.4 · p. 36

E.4 NanoChat

  • Within the first 24 active hours, NanoChat improves from 0.992692 to 0.905235 BPB (8.81%) across 1,465 successful evaluations.The 0.898671 best observation is an extension result, not a 24-hour result or a separately held-out test score.Source: §E.4 · p. 36
  • At the 24-hour cutoff, NanoChat’s incumbent has 1,723.9M parameters versus 50.3M initially and trains on 547.9M tokens versus the reference’s 728.2M.The comparison therefore is not architecture-only, equal-parameter, or equal-token; the 1,610.6M-parameter lexical table accounts for most of the incumbent’s capacity.Source: §E.4 · p. 36

E.5 Findings Across Research Tasks

  • NanoChat traces document reuse and revision of evidence, but do not establish that shared memory caused performance gains or identify each reused idea’s contribution.The run used experimenter/synthesizer and critic/verifier roles at a configured 3:1 ratio, supporting adaptive work selection within those roles rather than specialization without predefined roles.Source: §E.4 · p. 38, §E.5 · p. 39

E.4 NanoChat

  • NanoChat’s selected path combines branches and revisions, including reuse of failed trigram designs, but its extension endpoint is a best observation rather than a verified final architecture.The lowest observed extension score is 0.898671; an exact-source repeat scored 0.899864, and no completed finalizer submission was located.Source: §E.4 · pp. 37–38

F.1 Declared Reuse and Selected-Solution Ancestry

  • On ProgramBench, expanded informed_by= ancestry covers most of the contribution graph, while Git-parent ancestry alone accounts for roughly one quarter.Figure 7 reports coverage by budget; neither benchmark shows a consistent increase in coverage with budget, and proof ancestry can understate reuse because alternative proofs and critiques may inform selection without entering the selected proof’s ancestry.Source: §F.1 · p. 39

F.2 Persistent Artifact Production and Consumption

This appendix measures how workers produce and consume voluntary artifacts, combine work across branches, and shift their activity as runs progress. It also reports worker episode outcomes across the two benchmarks.

F.2 Persistent Artifact Production and Consumption

  • 32% of B100 and 62% of B500 ProgramBench runs produce human-readable findings; proof runs create bundles in 21, 20, 19, 23, and 20 of 30 tasks from B20 through B160.Proof computation and probe scripts account for 759 of 863 voluntary bundles.Source: §F.2 · p. 40, Figure 18 · p. 41, Table 11 · p. 41
  • Proof artifacts are frequently accessed but less often actively reused than ProgramBench artifacts; 22.6% of proof protocol revisions connect to the winner through a consuming contribution.Access and winner linkage are distinct outcomes: proof artifacts can support verification without entering the selected proof’s ancestry.Source: Figure 19 · p. 42, §F.2 · pp. 42–43

F.3 Collaboration Workflows

  • ProgramBench workflows include late integration of mature branches and gradual search-and-pivot, while the examples are not exhaustive or mutually exclusive strategies.The dominant pattern can change with compute budget.Source: §F.3 · p. 43

G.1 Worker-Activity Taxonomies, Activity, and Coordination-Tool Use over Time

  • After shared startup, ProgramBench sustains implementation, while proof runs shift toward selection, which occupies 64.8–82.2% of episodes in their final fifth.Proof workers use targeted Git reads rather than gs_explore or gs_inspect; ProgramBench dashboard and inspection calls rise near termination.Source: §G.1 · pp. 44–45, Figure 20 · p. 45, Figure 21 · p. 46

G.2 Worker Episode Outcomes and System Health

  • Roughly three quarters of ProgramBench episodes end in contribution commits across budgets; 615 of 20,000 charged B100 episodes produce no accepted action.The latter include 515 candidate-publication or required-artifact validation failures, 79 nomination-validation failures, and 21 invalid exits from API failures, turn limits, or protocol violations.Source: §G.2 · p. 45, Figure 22 · p. 46

G.3 Voting Dynamics and Selection Timing

This appendix examines when workers nominate candidates, how they abstain from redundant work and inspect shared history, and two execution and readout tradeoffs.

G.3 Voting Dynamics and Selection Timing

  • ProgramBench winners appear at 93.1–98.0% median progress, while proof winners appear at 8.6–17.5%; nominations arrive late on ProgramBench but early and accumulate on proof runs.Across-budget voting differences are descriptive because proof runs differ in configuration.Source: §G.3 · pp. 47–48, Figure 24 · p. 48

G.4 Voluntary Abstention and Redundancy

  • 94.9% of adjudicable ProgramBench abstentions were justified, while the proof audit confirmed cited evidence existed, not proof equivalence or correctness.ProgramBench abstainers consume about 6–7% of input and output tokens; proof abstainers consume 1.2–2.8% of input and 0.3–0.8% of output tokens.Source: §G.4 · p. 49
  • Declared redundancy stays near one worker in six on ProgramBench but below 2.7% on proof runs; proof abstainers mainly cite committed proofs, unlike ProgramBench’s in-flight overlap.Of 273 proof redundancy cases, 268 include committed evidence and only three cite in-flight work alone.Source: §G.4 · p. 48, Figure 25 · p. 50

G.5 Selective Inspection of Shared History

  • Workers inspect bounded candidate sets as repositories grow: proof history reach falls from 81.9% at B20 to 53.8% at B160, while ProgramBench look-back expands in commits.Workers use summaries and declared dependencies to focus on an integrated candidate and relevant alternatives; the tested budgets do not establish effectiveness at substantially larger graph sizes.Source: Figure 29 · p. 52, §G.5 · p. 53

G.6 Architectural Tradeoffs

  • Synchronous execution increases mean score by +1.05 points but raises mean task runtime by a factor of 1.30; the authors interpret asynchronous execution as reducing straggler-induced latency.The experiment does not establish statistical equivalence between schedules.Source: §G.6 · p. 53, Table 13 · p. 54
  • On the development set, a central selector raises mean score by 2.34 points and a central aggregator lowers it by 3.87 points versus decentralized nomination; limited seeds preclude a general advantage claim.The central aggregator also takes 16% longer; the comparisons do not isolate persistence, branchability, or semantic dependencies’ individual performance contributions.Source: §G.6 · p. 54, Table 13 · p. 54

H GitSwarm Worker Prompts

The appendix describes GitSwarm worker prompts and the auditable workflows they prescribe for advancing work in the shared Git graph. Across benchmark-solving and research tasks, the prompts define task-specific deliverables and operating rules, emphasizing inspection of prior work, bounded contributions with provenance, and evidence-based assessment.

H.1 Prompt Assembly

  • Each worker’s system prompt combines a global component with a task addendum; Gemini workers also receive a shell extension, and Codex workers a transport override.The user prompt is task-specific; the appendix lists configurations for ProgramBench, IMOProofBench-Advanced, and research runs.Source: §H.1 · p. 54, §H.2 · p. 54, Table 14 · p. 55

H.2 Global System Prompt

  • Workers are prompted to treat the repository as a discovery graph whose nodes can be stepping stones but must remain independently inspectable candidate answers.The global prompt instructs workers to inspect, test, criticize, repair, extend, combine, or replace prior nodes.Source: §H.2 · p. 55
  • Workers must inspect committed and in-flight work, refresh stale views, and judge candidates from direct evidence rather than nomination counts or popularity.The prompt describes Git tools and gs_inflight as ways to inspect durable contributions and ongoing activity.Source: §H.2 · p. 56
  • For a contribution, workers define a bounded gap, deliverable, success criterion, and abort condition, then abstain if the work is redundant or no longer worthwhile.They are told to prioritize expected marginal value given completed and active work.Source: §H.2 · p. 56
  • Workers choose one target commit as the physical parent and record every target commit as a semantic source in the final commit’s informed_by field.The prompt distinguishes the starting tree from semantic provenance and requires including the Git parent among the sources.Source: §H.2 · p. 57
  • A contribution must preserve candidate-answer structure, be validated as appropriate, and leave artifacts that make its reasoning, evidence, limitations, and next steps legible.Workers are instructed to create MY_THOUGHTS.md and an evidence-based EVALUATION.md before committing.Source: §H.2 · p. 57
  • In a separate nomination action, workers compare existing candidates without originating, repairing, or synthesizing solutions; valid nominations each count as one vote.The candidate with the most votes is selected at run end, with ties broken by lexicographic full commit SHA.Source: §H.2 · pp. 55, 57
  • Workers compare the full candidate graph using comparable evidence, keep nomination judgments independent, and abstain when no defensible or nonredundant contribution is available.Graph coverage uses ancestry, diffs, evaluations, and evidence to focus deeper inspection on serious contenders; explicit abstention does not consume a worker-budget slot.Source: §H.2 · pp. 57–58
  • A nomination branches from one exact existing candidate, preserves its substantive solution, and records the SHA, comparison evidence, rationale, limitations, and uncertainty.The pending nomination is scheduler-validated and published as an immutable ballot; the nomination metadata commit is not itself eligible for votes.Source: §H.2 · pp. 58–59
  • Each worker makes exactly one commit with verified `informed_by=` provenance for scoped source commits, then exits after the commit is detected.Nomination commits include the candidate set that materially informed the comparison and update `BEST_SOLUTION.md`.Source: §H.2 · pp. 58–59

H.3 Task Addenda

  • ProgramBench workers must produce an original codebase and `compile.sh` whose executable reproduces the reference behavior, which is tested against hidden cases.They may observe the execute-only reference during development but must not inspect its bytes; each candidate must remain runnable without the reference at grading.Source: §H.3 · p. 59
  • For IMO ProofBench, the selected candidate’s `SOLUTION.md` is the canonical scored artifact, and claimed complete answers must include the full proof there.An LLM judge scores the final answer on a 0–7 scale; presenting a wrong answer as correct scores 0.Source: §H.3 · pp. 59–60
  • RMT workers preserve the workbench, use protected evaluations for serious candidates, record evidence and uncertainty, and pursue actionable experiments while reopening collapsed searches.Candidates may preserve tested negative results; workers must cancel only with concrete evidence that work is stuck, invalid, or no longer useful, and submitted jobs must reach terminal status before exit.Source: §H.3 · p. 60

H.4 Model- and Transport-Specific Additions

  • Gemini-3.1-Pro workers must wait for command outputs before dependent shell commands, run `gs_fork` standalone, and use only permitted read-only or coordination commands before branching succeeds.The harness uses `set -e`, skips remaining blocks after a failure, and rejects other pre-branch commands; graph, scope, dependency, and nomination rules remain unchanged.Source: §H.4 · p. 60
  • Workers must issue one response per evidence-dependent step, inspect the returned output before choosing the next action, and combine commands only when they are independent of one another’s output.The prompt instructs workers to interact frequently rather than predict command results in advance.Source: §H.4 · pp. 60–61
  • Before branch confirmation, workers may only inspect or use coordination tools; they must fork as a standalone action and wait for confirmation before creating files or running validation.They must not combine forking with file writes or commits, create candidate files before branching, or mix contribution and nomination workflows.Source: §H.4 · pp. 60–61
  • ProgramBench Codex workers must perform every task operation through the guarded `pb.container_bash` MCP tool and stop immediately after a successful commit or recorded abstention.The tool runs commands in the worker’s private Git worktree inside the task’s network-disabled container.Source: §H.4 · p. 61

H.5 User Prompts

  • Workers must reproduce the documented behavior of a binary from its documentation and observed execution, without internet, preinstalled project dependencies, source-code discovery, wrapping, or runtime dependence on the reference.The binary is provided in the worktree; differential checks are investigative and validation tools, not the deliverable.Source: §H.5 · p. 61
  • Each graph-expansion node must retain a candidate submission structure, including a committed `compile.sh` that produces `./executable`, even when its contribution is exploratory or evaluative.Differential checks are investigative and validation tools rather than the deliverable.Source: §H.5 · p. 61
  • Workers are given a spawn-time snapshot of swarm status, but nomination counts do not indicate candidate quality, and candidates with zero nominations remain eligible for consideration.The prompt directs workers to use `gs_dashboard` for a current snapshot.Source: §H.5 · p. 62
  • IMOProofBench workers must provide a complete, rigorous proof graded from 0–7, explicitly justify every claim, handle every case, and organize the argument clearly.Each step must follow from the preceding step or a cited theorem.Source: §H.5 · p. 62

Related papers

Loading 2610.04862v1…