Source-linked AI summary
Agora: Git as Shared Memory for Collective AutoResearch
Yifan Zhang, Yunheng Zou, Shaokun Zhang, Jian Hu, Hao Zhang, Binfeng Xu, Jan Kautz, Yi Dong
TL;DR
Autonomous research agents lack durable shared state, causing repeated search and difficult verification across sessions. Agora addresses this with an append-only Git DAG that stores claims and provenance while exposing frontier, verification, and diversity signals. In its first sustained run, 13 agents improved frozen hybrid-model initialization from 3.39 to 1.899 bpb without training data, with 165 reproductions.
Problem
Autonomous research sessions lose their learning between runs, leaving agents unaware of failed experiments, abandoned branches, and verification needs.
Method
Agora stores research as an append-only Git DAG of immutable contributions and uses derived views plus diversity-aware attention to coordinate independently working agents.
Results
1.899 bpb was achieved from a random baseline of 3.39 in a 12-day run by 13 coding-agent sessions, with 165 reproductions.
Takeaways & Limitations
The run suggests that durable shared research state can support collective progress and reveal coordination dynamics, including concentration and neglected alternatives.
Takeaways & Limitations
The run does not establish causality because it lacked matched comparisons without Agora or with a plain leaderboard, and leaving the first basin required showing agents a map.
Abstract
from arXiv · showhide
Autonomous research loops such as AutoResearch show that one coding agent can improve a training setup unattended. Run several of them and each session starts from scratch, so more agents tend to mean more duplicated search rather than more discovery. Agora is a shared memory for such agents: research is recorded as an append-only directed acyclic graph (DAG) stored in Git, so that every claim is a commit anyone can check out and rerun. Each result, insight, hypothesis, verification, and report is an immutable commit whose parent edges say what it builds on; a derived index exposes the frontier, the neglected branches, and the verification status of each claim, and a diversity-aware selection rule keeps the community from collapsing onto one leader. We describe the system and report its first sustained use: a run of nearly 12 days in which 13 language-model workers, with no assigned tasks and no central planner, worked on a weight-transfer problem. Given 141 pretrained donor models and a frozen 119.6M-parameter attention-SSM hybrid whose dimensions match no donor, the workers had to initialize the target without training data or gradient updates. They published 1,703 contributions and drove the evaluator from 3.39 to 1.899 bits per byte, closing 62% of the gap to a trained GPT-2 124M. The winning recipe compresses donor next-token statistics into the target's embedding and output head, then adds a short-range context signal through sparse edits to attention, feed-forward, and state-space blocks. Its 145-commit ancestry spans 15 accounts, and 165 independent reproductions were posted, none of which failed. We describe the single mid-run human intervention that pulled the community out of a monoculture, what the trace does and does not establish, and the controlled comparison that would settle whether shared research state improves discovery per unit of compute.
1. Introduction
Agora addresses duplicated search in autonomous research by giving agents durable shared state: an append-only Git DAG that exposes claims, lineage, verification, and neglected alternatives. In a 12-day run, 13 coding-agent sessions used this shared graph to improve a frozen hybrid language model without training data, reaching 1.899 bpb from 3.39.
- 1. Introduction: The system exposes a public frontier, immutable lineage, negative results, independent verification, and neglected branches without imposing a single workflow.Its graph serves as a coordination and quality signal rather than deciding which claims are true.
- 1. Introduction: Agora records every research contribution as an immutable Git commit in an append-only DAG, with parent edges representing what each contribution builds on.A derived index provides searchable views, while the Git history is the only state exchanged among workers.
- 1. Introduction: 1.899 bpb was reached from a random baseline of 3.39 by 13 coding-agent sessions working for 12 days on frozen hybrid-model initialization without training data.The run used only a two-page brief, an evaluator, and the shared graph.
- 1. Introduction: The run produced 165 reproductions, and one human intervention showing agents a map of their concentration ended a five-day monoculture within a day.The intervention addressed concentration without assigning workers specific tasks.
- 1. Introduction: The paper contributes the DAG formulation, separation of storage from evidence and attention allocation, an implementation prototype, a sustained empirical run, and a matched preregisterable evaluation.The proposed evaluation is intended to test what the initial run leaves open.
2. Related Work
Related work spans collective intelligence, multi-agent coordination, autonomous research, reproducible computational artifacts, and exploration methods. Agora differs by preserving durable claim-level state across independently scheduled researchers while using heuristics to surface underexplored branches.
- Collective intelligence and scientific institutions: Scientific-institution research emphasizes public knowledge, independent discovery, interaction structure, and collaboration patterns as complements to individual ability.This frames Agora as infrastructure for collective rather than isolated research.
- LLM multi-agent systems: Prior multi-agent systems coordinate through role play, programmable conversations, procedures, orchestrators, or staged workflows, generally organizing interaction within a task.Agora instead focuses on shared research state that outlives individual workers.
- Autonomous research agents: Agora complements end-to-end research agents by preserving durable negative results, lineage, and verification across researchers scheduled independently.It prescribes no complete researcher workflow.
- Shared workspaces and reproducible artifacts: Reproducibility systems version code, data, dependencies, research objects, workflows, or experiment lifecycles, whereas Agora applies shared-state ideas one level up to claims.Its contribution DAG also exposes verification status, neglected branches, and attention signals.
- Exploration, open-ended search, and quality diversity: Exploration methods such as bandits, novelty search, MAP-Elites, and quality-diversity methods motivate Agora’s diversity-aware attention allocation.The related selection heuristics are intended to surface underexplored branches rather than guarantee optimal planning.
3. Agora: A Git-Backed Research DAG
Agora represents research as an append-only Git-backed DAG, making contributions, lineage, verification, and neglected alternatives visible without imposing a central workflow. Derived indexes and diversity-aware selection help participants coordinate while preserving project-specific evaluation and safety boundaries.
- Problem setting and design goals: The platform answers what was tried, which claims have support or conflict, where the frontier and neglected alternatives lie, and which lineage produced each result.These views are derived from the contribution graph and support searchable inspection of the research state.
- Problem setting and design goals: Agora is a coordination substrate rather than a lab manager: projects define their own instructions, metrics, artifact contracts, and safety boundaries.The prototype provides Git storage, SQLite-derived indexes, APIs, CLI commands, and web views, but does not assume one scoring rule fits every field.
- Contribution graph and provenance: Agora models project state as a directed acyclic graph whose Git commits store artifacts, claims, metadata, metrics, tags, and parent relationships.Git hashes and parentage make history append-only and acyclic, while code-bearing contributions retain the full repository state.
- Contribution graph and provenance: Agora separates immutable storage, downstream evidence, and attention allocation so projects can publish through shared interfaces without sharing filesystems, models, or conversations.Both metadata-only and code-bearing publication paths produce the same kind of graph node, while the index and views can be rebuilt from Git.
- Contribution graph and provenance: Its evidence score rewards qualifying work that other accounts reproduce or build on, excludes self-citation and failed verification, and does not itself establish truth.Acceptance still depends on each project's evaluator, controls, and artifact policy.
- Diversity-aware attention allocation: Because a leaderboard encourages convergence on one parent, Agora combines exploitation with known-cluster and novel-cluster exploration using diversity-aware candidate ranking.The split exposes the trade-off between refining leaders, extending thin clusters, and inspecting untouched alternatives.
4. The Weight-Transfer Run
The run tested whether independently scheduled coding agents could use Agora to initialize a mismatched hybrid language model from pretrained donors without target training data or gradient updates. Over nearly 12 days, the community produced a sustained stream of contributions under a no-assigned-task, no-central-planner protocol.
- Research loop: The report attributes the produced methods and reproductions to the agents, while the authors defined the task and evaluator and checked the published record.The checking procedure is described separately in the paper.
- Problem setting and design goals: The task was to recover predictive quality in a 119,572,320-parameter, 14-layer attention-SSM target whose dimensions matched no donor, using only donor weights and forward passes.The donor zoo contained 141 open-weight models from 32 architecture families, and target training, fine-tuning, and evaluator modification were forbidden.
- Research loop: Thirteen worker accounts contributed 1,699 of the 1,703 records, using fresh checkouts, independent evaluations, and a loop of analysis, parent selection, modification, publication, and renewed analysis.Workers had access to the CLI, Git, PyTorch, Transformers, donor models, the evaluator, and one 80 GB GPU.
- Research loop: The community sustained roughly 170 contributions per day once all 13 workers were running, while explicit negative-result and explore-novel tags appeared only after the landscape and diversity views were deployed.The publication trace therefore records both activity volume and a change in the available coordination signals.
- Research loop: 1,703 contributions were published across 11 days and 19 hours, comprising 1,124 scored results, 284 insights, 203 hypotheses, 165 verifications, and one report.The counts have tag overlaps, and 233 scored results established a new best.
4.4. The winning recipe
The winning recipe transfers donor predictive behavior rather than donor parameters: it compresses next-token statistics into the target embedding and output head, then restores short-range context through sparse structured edits. The committed method was assembled as a sequence of evaluated changes on the current best.
- The winning recipe: Each constant in the final recipe was introduced as one change on the then-current best and retained because the evaluator improved.The cutoff method is represented by an 83-module ancestry traced to its root and checked against the committed code.
- The winning recipe: Stage A queries six vocabulary-compatible donors under 28 contexts, blends their next-token log-softmaxes, and factorizes the resulting transition prior into the target embedding and output head.Donor weights and context weights produce a 50,257-token transition representation before randomized SVD supplies the target-sized factors.
- The winning recipe: Stage B restores context with sparse deterministic edits on 96-dimensional hidden-state bands assigned across attention, feed-forward, and state-space blocks.Attention performs scaled causal mean-pooling, layer 0 receives a projected GPT-2 small MLP slice, and SSM blocks use gated depthwise causal convolutions.
- The winning recipe: The algorithm initializes the target by setting all sublayer weights to zero, using identity norms, and then applying the Stage-A factors and Stage-B routes.The committed pseudocode specifies the donor transition prior, SVD factorization, attention routing, MLP projection, and SSM kernels.
4.5. Evidence
Agora’s evidence trace shows rapid gains from statistical priors, later refinements, and extensive cross-account reproduction, while documenting negative results and evaluator limits.
- Trajectory: Figure 3 places every scored contribution on a timestamped log-bpb axis, showing the main reduction from early statistical priors and first sub-1.90 scores after the May 2 landscape views.The rendering covers the first 1,703 contributions through May 8; colors encode score bands, parent edges show lineage, and orange rings identify leaders after May 2.
- Trajectory: The first 18 scored contributions produced about 98% of the total reduction, moving from failed parameter copying to unigram and bigram donor statistics.The remaining 1,106 scored contributions found the next 0.03 through additional donors, longer contexts, improved SVD, and re-enabled model sublayers.
- Primary result: Table 4’s milestones are sequential search stages selected on one development evaluator, not controlled ablations.Adjacent rows therefore document the winning ancestry’s progression rather than isolated causal effects.
- Negative results: Negative-result records ruled out 48-prefix expansion, flattened singular values, native Mamba transplantation, direct GPT-2 embeddings, and mismatched-tokenizer priors.The window contains 53 contributions explicitly tagged as negative results, which later workers used when selecting directions.
- Lineage and reproduction: The winning ancestry contains 145 commits from 15 accounts, with 115 of 144 parent edges crossing account boundaries; 165 verifications reported no failures.Forty of the winner’s 144 scored ancestors were independently reproduced, while cross-hardware scores differed by up to 1.3 × 10−3 bpb within tolerance.
- Primary result: 1.899044 bpb was achieved without training data or target gradients, versus 3.3923 for random initialization and about 1.0 for trained GPT-2 124M, closing 62% of the gap.Because all components were selected on the same 200-text development evaluator, the improvement from 3.39 to about 1.90 is more meaningful than the final decimal places.
4.6. Coordination dynamics
The trace shows rapid early exploitation, concentration on a narrow leader lineage, frequent parallel rediscovery, and a shared diagnosis of a performance ceiling. A mid-run diversity intervention followed a stalled, monocultural phase, while verification recomputed the reported statistics from exported artifacts.
- One narrow lineage collected most follow-on work, with side branches short and quickly abandoned.Figure 4 presents the full graph and highlights the eventual leader’s ancestry.
- The first eight improvements produced roughly 70% of the total descent, while the first 18 scored contributions produced about 98%.
- Of 696 equal-score pairs from different accounts, 63% appeared within an hour and 80% within six hours.Figure 5 describes these time gaps and frontier-parent adoption.
- Several contribution families clustered near 1.90 bpb, alongside an agent-generated explanation that the evaluator was globally linear and target sublayers underused.The paper reports these interpretive claims as the agents’ claims rather than established conclusions.
- After analysis views showed over a third of activity in one semantic cluster and a stalled leaderboard, researchers deployed clustering, diversity summaries, and diversity-aware UCB.Workers left the monoculture within a day of seeing a map of their concentration.
- The complete contribution stream and graph were exported, and every count, statistic, and figure was recomputed from that export rather than agent summaries or the leaderboard.Reported best scores were also checked algebraically and against reproductions.
5. Conclusion
Agora addresses duplicated search by making research contributions durable, addressable, and linked through Git-backed provenance, derived indexing, and diversity-aware attention. Its first sustained run achieved a strong initialization result and repeated reproduction, but the causal benefit of shared research state remains untested.
- Agora offers Git-backed provenance, a derived index, and diversity-aware attention to make contributions durable, addressable, and linked to their dependencies.
- Thirteen agents found a no-training initialization that closed 62% of the gap to a trained model and reproduced one another’s results 165 times.The run also produced a shared diagnosis of its performance ceiling.
- The run did not establish causality because it lacked a matched comparison using the same models and compute without Agora or with a plain leaderboard.The proposed next step is a matched comparison measuring when a research DAG improves discovery versus organizing parallel waste.
A. Reproducibility Requirements
A retained community run must preserve immutable identities for the research environment, participants, and full contribution history so the record remains reproducible.
- The run should retain immutable identities for project policies, software revisions, dependencies, models, data, architecture, hardware, participants, prompts, configurations, seeds, concurrency, and compute budgets.
- It should also preserve the full contribution DAG with canonical hashes.
B. Minimal Contribution Record
A minimal contribution record combines canonical server metadata with artifacts and metrics sufficient to identify what was published and reproduce a verification.
- The canonical record includes the project, agent account, parent hashes, tags, description, metric value, and durable run-manifest reference.
- A verification additionally records the target hash, reproduction configuration, evaluator identity, and reproduced value.Without these artifacts, a verdict is only a coordination hint rather than strong validation evidence.
C. Proposed Matched Evaluation Matrix
The proposed evaluation compares communities under matched agents, models, compute, evaluator, and wall-clock budgets. The analysis treats the entire community run as the primary unit rather than treating individual commits as independent samples.
- Matched agents, models, compute, evaluator, and wall-clock budget define the minimum community-level comparison.
- The entire community run is the primary analysis unit, while commit-level observations serve only as diagnostics and are not independent samples.