Source-linked AI summary
A Vocabulary for Multi-Agent Automated Research Systems
Bardiya Akhbari
TL;DR
Multi-agent autoresearch systems vary across many interacting design axes, making comparisons ambiguous. This paper introduces a vocabulary that decomposes those choices and illustrates its coverage on recent systems, distinguishing agent count from architecture-task alignment and separating generative from evaluative taste.
Problem
Comparisons among multi-agent systems are ambiguous because designs change communication, memory, initialization, capabilities, and evaluation simultaneously.
Method
The paper defines a vocabulary covering agents, operations, capabilities, communication, shared state, control, initialization, trajectories, and evaluation.
Results
The vocabulary describes and distinguishes recently published autoresearch systems, while showing that architecture-task alignment predicts gains more than agent count.
Takeaways & Limitations
The framework makes structural comparisons and design changes testable by isolating coordinates and distinguishing generative taste from evaluative taste.
Takeaways & Limitations
True quality q is generally uncomputable, so evaluative taste remains defined relative to a proxy-quality gap rather than directly measured.
Abstract
from arXiv · showhide
We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare. The vocabulary specifies 1) who the agents are, 2) what operations are available in the system, 3) who may invoke them, 4) how agents communicate, 5) what information is visible within and across runs, 6) how the next action is chosen, 7) how a run begins, and 8) how outputs are evaluated. A trajectory records one run from the input task to the returned artifact. Because agents, operations, and initialization may be stochastic, repeated runs on the same task induce a distribution over trajectories rather than a single behavior. Our vocabulary turns structural design questions, such as when agents should communicate, gain or lose a capability, or carry information across runs, into testable choices. It also makes the evaluator a component of the system, since reported gains depend on how closely the proxy score matches true quality. That separation also splits the vague complaint that these systems lack taste into two failures with different solutions. Generative taste is the rate at which a system proposes novel trajectories before any score is observed, and evaluative taste is the gap between the proxy score and the quality it should match. We instantiate the vocabulary on recent autoresearch systems to illustrate that it covers designs that differ widely in structure.
1 Introduction
The paper introduces a vocabulary that decomposes multi-agent automated research systems into explicit design coordinates, enabling controlled comparisons and isolating the sources of reported improvements. It formalizes these systems as a tuple covering agents, operations, capabilities, communication, state, control, initialization, and evaluation.
- Motivation: End-to-end research systems automate literature review, idea generation, coding, experimentation, interpretation, and iteration, often as search over research trajectories.Examples include evolutionary search over candidate experiments and supervisor–researcher optimization of inference-system designs.
- Motivation: Comparisons between systems are ambiguous because communication, memory, initialization, and evaluation may change simultaneously.The vocabulary decomposes systems into coordinates fine-grained enough to isolate individual design choices.
- System vocabulary: The formal tuple records agents, operations, communication, capabilities, shared state, control policy, initialization, and evaluator.In the notation, A denotes agents, O operations, C communication, α capabilities, S shared state, π control, ι initialization, and e evaluation.
- Contribution: Varying one coordinate at a time helps identify whether improvements arise from communication, initialization, memory, evaluator integrity, or capability assignment rather than from multi-agent structure alone.Treating the evaluator as a coordinate also provides a handle on the previously informal notion of taste.
- Contribution: The contribution is a common language for describing systems and identifying which coordinate each design changes, supporting analysis of trajectories, optimization, current systems, and open design questions.The paper develops the vocabulary across Sections 2–7 and surveys related work in Appendix A.
2 Problem specification
A problem specifies the tasks a system faces, the artifacts that count as solutions, and how solution quality is measured. It also includes task distributions, measurable task features, ideal and proxy evaluation, and resource budgets, with recursive self-improvement as a concrete instance.
- Problem components: A problem maps each task x ∈ X to a solution artifact y ∈ Y and defines how that artifact’s quality is measured.Examples include research questions yielding papers, bug reports yielding patches, clinical cases yielding diagnoses, and code targets yielding optimized programs.
- Problem components: Task features ϕ(x) represent measurable properties that group tasks into families for analysis.Features can distinguish tasks by subtask count and a fixed reference agent’s performance on similar tasks.
- Problem components: True quality q evaluates solutions across k objective dimensions, scalarization ω ranks them, and proxy-quality gap measures the discrepancy from a computable evaluator.True quality is generally uncomputable; objective dimensions can include correctness, novelty, and cost.
- Problem components: The task distribution D specifies how inputs are sampled, while each task has a hard budget measured in tokens, dollars, wall-clock time, and/or operations.AIRS-Bench exemplifies a fixed benchmark distribution with 20 machine-learning tasks, held-out test labels, and a programmatic scorer.
- Recursive self-improvement: Recursive self-improvement treats a complete AI system as the task and a more capable successor as the solution.Examples rewrite a system’s harness or apply a code-improvement program to itself, retaining changes that improve benchmark scores.
3 System specification
The vocabulary specifies multi-agent research systems through separable choices about agents, operations, communication, state, control, and evaluation. It distinguishes within-run refinement from cross-run improvement and separates communication topology shape from routing control.
- Agents: Agents are stateful entities with private memory, while stateless calls with fixed roles are modules rather than agents.An agent’s backbone, private memory, initial memory, and role define its identity; private memory persists across turns but remains hidden from other agents.
- Communication: Communication separates edge shape from control: Λ determines which directed links exist, while πroute determines who selects the active edge.The same edge set can operate under centralized, hierarchical, or decentralized control, so labels such as “hierarchical” or “star” are incomplete alone.
- State: Shared state is scoped as within-run Sbtw, external Sworld, or cross-run Scross, with cross-run state enabling accumulated improvement across executions.Cross-run state includes skill libraries, population databases, and distilled stores; systems with Scross = ∅ can still improve within a run but restart from scratch.
- Control: Control decomposes into routing, stopping, meta-control, and exploration, with meta-control able to alter agents, capabilities, communication, state, or the control policy during a run.Routing must select an operation permitted by the acting agent’s capability assignment, whereas meta-control changes system structure rather than merely choosing the next action.
- Evaluation: Evaluation specifies metric type, scoring mechanism, and integrity protections that limit drift between the evaluator’s score e and target quality q.Protections include judge decoupling, sandbox isolation, metric-channel blackout, and contamination checks.
4 Trajectory
A trajectory is one run under specification M, recording agent actions, observations, shared and private state, structural changes, and cost through a horizon. It yields a final artifact for downstream evaluation while distinguishing object-level, communication, and structural actions.
- Trajectory definition: A trajectory is one run under M, measured over its steps by cost, trajectory distribution, and coordinate comparisons rather than the final artifact alone.Scoring the returned result is performed separately downstream by evaluator e.
- Trajectory contents: At each step, the trajectory records the acting agent, action, observation, shared state, and private memory, plus structural changes when πmeta rewrites them.The recorded structural triple is (At, αt, Λt): agent set, capability assignment, and communication space.
- Action structure: The action space separates object-level work from communication and structural change, making communication, spawning, and capability changes explicit policy-controlled choices.Operation calls act on Sworld through the acting agent’s capability assignment; messages use permitted communication edges and write to shared intermediary state and recipient queues.
- Returned artifact: The run extracts a final solution y(τ), such as committed code or an announced diagnosis, while retaining the trajectory-level record for analysis.The final artifact is distinct from the steps used to produce it.
- Cost accounting: Trajectory cost sums action costs, with operation calls charged by API or sandbox invocation, messages by tokens and processing, and structural actions by setup.The halt action has zero cost, and separating message and structural costs supports evaluating whether their expected gain in e(τ) justifies them.
5 Proxy optimization
Proxy optimization separates search policy from evaluator quality: systems maximize a proxy over reachable trajectories, while deeper selection can increase divergence from true quality. This framing treats reward hacking and evaluation integrity as evaluator risks requiring targeted protections.
- Objective: The autoresearch objective separates policy search from evaluator calibration, so observed gains may reflect either better exploration or a proxy farther from true quality.The optimization writes π and e as separate terms: search under a budget and the proxy-to-quality gap.
- Search: A system returns the highest-scoring trajectory from its budget-feasible support, approximating an intractable argmax through methods such as Best-of-N, buffers, migration, or restarts.These methods differ in their bias-variance tradeoffs because the feasible trajectory set is too large to enumerate.
- Search: Communication, agent-set changes, capability grants or revocations, and cross-run information persistence are policy choices, so architectural-looking gains may instead come from more search.The same agents and tools can produce different outcomes under different search procedures.
- Evaluator risk: Selecting among more trajectories raises the overfitting tax because search increasingly favors rare cases where the proxy overstates true quality.This applies to argmax selection over N trajectories and parent selection in evolutionary loops.
- Evaluator risk: Reward hacking is measurable: MLR-Bench found fabricated or invalidated results in 8 of 10 audited coding-agent tasks, while METR cataloged 103 unprompted constraint violations.The cited exploit modes include seed cherry-picking, held-out-label exfiltration, direct test execution, and shortcut identification.
- Mitigations: Mitigations should strengthen the protection each exploit defeats, including variance control, metric-channel blackout, sandbox isolation, and contamination control.Constraining other system coordinates, such as revoking test-execution tools through α, can protect evaluator integrity.
6 Case studies
The case studies map recent autoresearch systems onto the vocabulary’s structural coordinates, distinguishing deliberate design choices from generic harness defaults. They span evolutionary populations, agent dialogue, shared forums, sequential workflows, and cross-run memory.
- Case-study method: Each case study records which vocabulary coordinates a paper deliberately sets and which remain generic, while holding the benchmark task distribution fixed.A coordinate counts as set when the paper motivates, ablates, or builds its contribution around that choice.
- Evolutionary systems: AIRA2 uses ephemeral uniform workers and evolutionary parent selection, whereas AlphaEvolve adds heterogeneous backbones and meta-control to balance candidate breadth with stronger generation.AIRA2’s main system uses n = 8 workers; AlphaEvolve combines Gemini 2.0 Flash with a stronger backbone.
- Structural patterns: The systems exhibit distinct structural patterns, including evolutionary populations, island migration, two-agent dialogue, seeded research directions, and other coordination designs.The figure summarizes these shapes without implying that every marked coordinate improves performance.
- Communication and capabilities: Glia uses a fixed Researcher–Supervisor pair sharing an execution context, while AAR uses nine parallel agents whose findings are broadcast through a shared forum.Glia assigns tools only to the Researcher; AAR’s forum lets agents browse synchronized findings locally rather than query a remote store by keyword.
- Workflow and memory: The case studies also include single-manager sequential search, role-specialized software teams, a negative control without cross-run memory, and Engram’s cross-run store.SimpleTES isolates the test-time scaling loop, while Engram inherits Glia’s problem layer and adds cross-run state.
7 Discussion
The discussion presents the vocabulary as a framework for distinguishing autoresearch-system designs and turning structural choices into controlled empirical tests. It also separates generative from evaluative taste and identifies capability assignment, exploration, policy, and task-distribution changes as under-explored directions.
- Vocabulary and empirical scope: The vocabulary describes and distinguishes recently published multi-agent autoresearch systems while extending to broader architectural designs.It represents choices about agents, operations, invocation, communication, visibility, action selection, initialization, and evaluation.
- Empirical design implications: Architecture-task alignment, rather than bare agent count, predicts gains, while topologies without a centralizing verifier sharply amplify errors.Existing studies commonly confound |A| with population size, island count, or fixed agent counts.
- Generative and evaluative taste: When the evaluator is miscalibrated, harder search can exploit high proxy scores while returning low-quality trajectories, so evaluator integrity should precede crediting stronger generation.The resulting overfitting tax makes gains difficult to attribute unless proposals are judged for novelty and quality independently of the evaluator.
- Open experimental coordinates: Capability assignment is the clearest near-term test: dynamic α remains untested because practical permission infrastructure is costly, not because dynamic control lacks justification.Glia provides a concrete static separation, with Supervisor α(Sup) = ∅, preserving independent judgment by preventing the advisor from performing worker tasks.
- Open experimental coordinates: Exploration η remains at its default, policy rewriting is exercised only by EvoX, and no system here changes the task distribution D.AAR injects diversity through initialization ι rather than η, while prior work explored environment-agent co-evolution.
A Related work
The vocabulary builds on classical multi-agent systems, MARL/Dec-POMDPs, LLM-agent surveys, autoresearch systems, and reward-hacking research while adapting their concepts to compare LLM-agent designs. It distinguishes structural comparison from workflow taxonomies and frames proxy evaluation as a central design concern.
- Classical multi-agent systems: Classical multi-agent systems provide the structural core of agents, communication, and shared state, while the vocabulary retains private memory, communication, and control policy for LLM agents.The paper notes that LLM-agent settings force departures from classical frameworks.
- Multi-agent reinforcement learning and Dec-POMDPs: MARL and Dec-POMDPs motivate viewing runs as trajectories, using stationary control policies, and optimizing expected cumulative reward over a budget.The vocabulary deliberately does not borrow three other elements from those frameworks.
- LLM-agent harnesses and surveys: LLM-agent harnesses and surveys catalog design patterns and workflow dimensions, whereas this vocabulary compares agent structure and coordination directly.The cited surveys cover systems such as AutoGen, MetaGPT, CAMEL, ChatDev, AgentVerse, and Voyager, while Tie et al. organize autoresearch by autonomy, workflow, and evaluation dimensions.
- Autoresearch systems: Autoresearch systems commonly describe research threads, branches, memory, and correctness audits functionally in prose, so the tuple labels their design choices in shared, comparable terms.The paper places its detailed examples within the broader autoresearch literature.
- Reward hacking and Goodhart’s law: Reward-hacking and Goodhart’s-law research motivates treating optimization against a proxy as distinct from achieving the quality that proxy should match.The cited literature includes reward hacking, specification gaming, and multiple Goodhart variants.