PaperlayerView exact revision on arXiv

Source-linked AI summary

ABSTRAL: Automatic Design of Multi-Agent Systems Through Iterative Refinement and Topology Optimization

Weijia Song, Jiashu Yue, Zhe Pang

arXiv:2603.22791v1cs.AI

TL;DR

Multi-agent system design lacks a human-readable account of which architectural patterns work for which problems and why. ABSTRAL iteratively refines a natural-language design document using execution traces. It reports measurable coordination costs alongside transferable design knowledge and specialist-role discovery.

  • Problem

    Existing automated approaches store intermediate knowledge as code, prompts, or skill libraries, but do not capture human-readable rationale about which architectural patterns work for which problems and why.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    ABSTRAL treats a structured natural-language design document as the optimization target and iteratively refines it using execution traces, including for specialist-role discovery.

    Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 4

  • Results

    26% turn efficiency and 66% of tasks exhausting fixed turn budgets quantify coordination costs, while the paper reports transferable design knowledge and specialist-role discovery absent from initial designs.

    Source: §1 · p. 1, §6 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    The converged SKILL.md document is inspectable design rationale that practitioners can selectively preserve or clear to support controlled knowledge reuse across domains.

    Source: §6 · p. 9

    All reported results come from single experimental runs, and a second SOPBench bank run peaked at 65% versus 70%, indicating meaningful run-to-run variance.

    Source: §7 · p. 10

Abstract

from arXiv · show

How should multi-agent systems be designed, and can that design knowledge be captured in a form that is inspectable, revisable, and transferable? We introduce ABSTRAL, a framework that treats MAS architecture as an evolving natural-language document, an artifact refined through contrastive trace analysis. Three findings emerge. First, we provide a precise measurement of the multi-agent coordination tax: under fixed turn budgets, ensembles achieve only 26% turn efficiency, with 66% of tasks exhausting the limit, yet still improve over single-agent baselines by discovering parallelizable task decompositions. Second, design knowledge encoded in documents transfers: topology reasoning and role templates learned on one domain provide a head start on new domains, with transferred seeds matching coldstart iteration 3 performance in a single iteration. Third, contrastive trace analysis discovers specialist roles absent from any initial design, a capability no prior system demonstrates. On SOPBench (134 bank tasks, deterministic oracle), ABSTRAL reaches 70% validation / 65.96% test pass rate with a GPT-4o backbone. We release the converged documents as inspectable design rationale.

1 Introduction

ABSTRAL treats multi-agent design knowledge as an inspectable natural-language document refined through execution-trace analysis. Its findings address coordination costs, transfer across domains, and specialist-role discovery.

1 Introduction

  • Existing automated approaches store intermediate knowledge as code, prompts, or skill libraries, but do not capture human-readable rationale linking architectural patterns to problems.The stated gap is knowledge about which patterns work for which problems, and why.Source: §1 · p. 1
  • ABSTRAL iteratively refines a natural-language design document by analyzing execution traces.The document is the optimization target, and a three-layer architecture supports trace-driven refinement, consolidation, and topology repulsion.Source: §1 · pp. 1–2
  • 26% turn efficiency and 66% of tasks exhausting fixed turn budgets quantify the coordination tax, while ensembles improve over single-agent baselines on tasks admitting parallelizable decomposition.The paper also reports that transferred design documents match cold-start iteration-3 performance in one iteration, and that contrastive trace analysis discovers specialist roles absent from initial designs.Source: §1 · p. 1
  • ABSTRAL contributes an inspectable, transferable SKILL.md design document, quantitative coordination-tax analysis, and trace-driven discovery of specialist roles.The authors describe the role-discovery capability as creating specialists absent from initial designs.Source: §1 · p. 2

2 Related Work

Prior approaches automate agent design or encode skills, but the supplied related work identifies gaps in topology search, role discovery, and inspectable design rationale. ABSTRAL is positioned as addressing these gaps through design knowledge stored in natural-language documents.

2 Related Work

  • Earlier skill paradigms store executable code or behavioral prompts, while skill-as-design-knowledge uses documents; ABSTRAL documents how to construct agent systems.The distinction is between documenting system construction and documenting task execution within a system.Source: §2 · p. 2
  • Existing structured skill documents describe how individual agents execute tasks, while SkillOrchestra uses routing descriptors for a fixed pool and does not discover roles or topology families.The supplied comparison also notes that SkillsBench found self-generated skills yield no net benefit without systematic curation.Source: §2 · p. 2
  • Prior automated design systems vary in their search strategies, while fixed-topology frameworks define collaboration protocols without searching topology.The related work contrasts code, neural parameters, and scalar scores with ABSTRAL’s readable design rationale, topology variation, and role discovery.Source: §2 · p. 2
  • Prior approaches optimize within fixed design regions, lack criteria for topology diversity and genuine role novelty, and propagate scalar feedback rather than causal knowledge about failures.The paper frames topology search, role composition, and trace interpretation as two open problems plus an orthogonal limitation of trace-driven approaches.Source: §2 · p. 2
  • SOPBench evaluates standard-operating-procedure compliance with a deterministic oracle, and published baselines range from 33.58% to 76.87%.Prior work cited in the passage found 41–86% of multi-agent failures to be structural.Source: §2 · p. 3

3 The ABSTRAL Framework

ABSTRAL makes the skill document the only artifact carried across its continuous design loop. Its inner loop builds, runs, analyzes, and updates systems; consolidation manages drift, while the outer loop explores structurally different topologies.

3 The ABSTRAL Framework

  • Only the skill document A_t crosses ABSTRAL’s continuous design-loop boundaries.The framework separates iterative refinement from consolidation and topology-family exploration.Source: §3 · p. 3

3.1 The Skill Document

  • A_t is a structured Markdown file with four sections: domain knowledge (K), topology reasoning (R), discovered role templates (T), and construction protocol (P).Each section is updated in response to its corresponding evidence class: EC1 through EC5.Source: §3.1 · p. 3

3.2 Inner Loop: Trace-Driven Refinement

  • Each inner-loop iteration builds an agent specification, runs it on a stratified sandbox task sample, classifies paired failed and successful traces, and updates the responsible document section.EC3 can trigger creation of a specialist role template in T for an agent handling incompatible subtasks.Source: §3.2 · pp. 3–4, Figure 1 · p. 4

3.3 Convergence and Consolidation

  • Convergence signals govern inner-loop termination, while periodic consolidation merges redundant rules, removes weakly supported ones, and elevates heuristics into principles.The consolidation quality metric targets a post-to-pre rule ratio of ρ ≤ 0.65.Source: §3.3 · pp. 4–5

3.4 Outer Loop: Structural Diversity

  • The outer loop explores qualitatively different design regions by requiring both graph-edit and role-set embedding distances, then preserving K while clearing T for re-derivation.The thresholds are graph edit distance ≥3 and cosine distance ≥0.25.Source: §3.4 · p. 5

4 Experiments

On SOPBench, ABSTRAL combines trace-driven refinement with topology exploration, reaching 70% validation and 65.96% held-out test pass rates. The experiments examine setup, outer-loop effects, specialist discovery, and ablations.

4.1 Setup

  • Under the fixed 20-turn budget, ensembles averaged 26% turn efficiency and 66% of tasks exhausted the limit, despite improvements over single-agent baselines.SOPBench contains 134 bank tasks with deterministic oracle evaluation; the search used 20-task validation batches and evaluated the selected configuration once on 94 held-out tasks.Source: §4.1 · p. 5

4.2 Main Results (SOPBench Bank)

  • 70% validation and 65.96% held-out test pass rates were achieved by the O3/I4, 6-agent configuration; the comparison to published baselines is confounded by framework overhead.A second run peaked at 65%, and the inner-only O1 ablation plateaued at 55%.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 What Does the Outer Loop Add?

  • The outer loop raises pass rate from 55% to 70% through topology-family changes, more domain-specific roles, and retained knowledge across loops.O1’s star topology plateaus at 55%; GED repulsion moves O2 to an ensemble family, and O3 refines role composition while preserving K.Source: §4.3 · p. 6

4.4 Specialist Role Discovery (EC3)

  • EC3 discovery progresses from broad functional roles in O1 to fine-grained specialists with epistemic stances, including Verifier and Oracle, in O3.Table 5 presents the specialist roles discovered across outer loops.Source: §4.4 · p. 7, Table 5 · p. 8

4.5 Ablations

  • The full pipeline reaches 70%, 15 points above inner-only O1 at 55%, with the outer loop contributing the remaining gain through topology transitions.The inner loop contributes 20 points from 35% to 55%; its gains plateau after 4–5 iterations per outer loop, while O2/I1 reaches 50%.Source: §4.5 · p. 7
  • Transferred knowledge lets O2/I1 reach 50%, matching O1’s I3 performance in one iteration, while O3/I1 reaches 55%.K is preserved and T cleared between outer loops; without K transfer, each loop would need about three iterations to rediscover basic domain rules.Source: §4.5 · p. 7
  • Consolidation at I6 temporarily disrupts pass rates as useful-but-redundant rules are removed, with recovery within 1–2 iterations.Without consolidation, rule counts grow monotonically to 83–84 by I5 in each outer loop.Source: §4.5 · p. 8

5 Analysis

The analysis traces how the design document and evidence classes change across iterations. It identifies topology failures as a structural bottleneck and reports consolidation trade-offs and search cost.

What does the skill document learn?

  • The document accumulates domain rules and conditional topology reasoning across outer loops, while role templates are cleared and rapidly repopulated.Examples include verifying account ownership before balance modification and preferring fan-out for tasks with independent sub-operations.Source: §5 · p. 8

Evidence class distribution

  • EC2 topology failures are most frequent overall at 51 events, including 42 in O2–O3 versus 9 in O1; EC1 domain reasoning errors total 28.EC3 missing-specialist events total 9, EC4 interface mismatches decline from 17 in O1 to 8 in O2 and disappear in O3, and EC5 appears in 6 iterations.Source: §5 · p. 8, Table 6 · p. 9

Why topology dominates within-family refinement

  • The O1 hierarchical topology plateaus at 55% because its manager bottleneck drives recurring EC4 failures that domain-knowledge edits cannot fix.EC4 falls to 8 total in O2 and disappears in O3 after the forced transition to an ensemble topology.Source: §5 · pp. 8–9

Consolidation dynamics

  • Consolidation fires at I6 in each outer loop with ρ ∈[0.63, 0.73] (targeting ≤0.65), temporarily lowering pass rates; recovery occurs within 1–2 iterations.The dip is most pronounced in O2 (0.60 →0.45) and O3 (0.55 →0.45).Source: §5 · p. 9

Search cost and efficiency

  • The 24-iteration search used 154K GPT-4o backbone tokens and 480 API calls, costing about $5 in backbone inference and taking 28.4 hours.RUN consumed 79% of wall-clock time; BUILD, ANALYZE, and UPDATE together consumed 21%.Source: §5 · p. 9

6 Discussion

ABSTRAL measures when multi-agent coordination is worth its turn cost and treats design knowledge as an inspectable document. Its convergence signals did not end any of the three outer loops.

When is multi-agent coordination worth the cost?

  • Under SOPBench’s 20-turn budget, ensembles achieve 26% turn efficiency and 66% of tasks exhaust the limit; fan-out helps parallelizable tasks but hurts long sequential chains.Successful tasks use concurrent specialist execution to compress tool calls into fewer turns, while routing overhead leaves insufficient turns for some sequential tasks.Source: §6 · p. 9

Why natural language works as an optimization target

  • The SKILL.md document is inspectable and optimizable, recording topology reasoning, domain rules, and specialist roles while allowing sections to be preserved or cleared for transfer.The document supports selective reuse, preserving K for domain knowledge or clearing T for role templates.Source: §6 · p. 9

Convergence behavior

  • All three outer loops ended at the hard cap Tmax=8 rather than convergence signals, whose thresholds may be too conservative for 20-task batches with ±10% pass-rate fluctuations.Only O2/I5 triggered the pass-rate-plateau signal, and subsequent improvement at I7 overrode it.Source: §6 · p. 10

7 Conclusion

ABSTRAL treats natural-language design documents as an optimization target and reports findings on coordination costs, knowledge transfer, and specialist discovery. Its conclusion also describes topology-distance methods and study limitations.

7 Conclusion

  • The SKILL.md document serves simultaneously as the search space, output artifact, and human-readable audit trail for MAS design.The paper contrasts these properties with code-based or neural representations.Source: §7 · p. 10
  • The three reported findings are 26% turn efficiency under fixed budgets, transfer that matches cold-start iteration-3 performance in one iteration, and discovery of specialists absent from initial designs.The paper describes the coordination tax as structural to multi-agent coordination under fixed budgets.Source: §7 · p. 10
  • ABSTRAL reaches 70% validation and 65.96% test pass rate on SOPBench bank with 134 tasks and a deterministic oracle.The paper presents these benchmark results as context rather than its primary claim.Source: §7 · p. 10
  • The authors flag single-run variance, possible gaps in the seven-category topology taxonomy, and a confound in the framework-overhead comparison.Future work includes adaptive turn allocation, cross-domain transfer at scale, and stronger backbones.Source: §7 · p. 10

A Formal Topology Distance

  • Converged topologies are role-labeled directed graphs; structural distinction uses GED ≥ 3, while semantic distance uses canonicalized role-set embeddings with cosine distance ≥ 0.25.The design landscape is projected to 2D with MDS and UMAP refinement for visualization.Source: A Formal Topology Distance · p. 12

B Convergence Signals

This section identifies convergence detector signals and gives the termination conditions used in the experiments.

B Convergence Signals

C EC3 Role Discovery: Full Specification

The appendix specifies when contrastive traces trigger role discovery, describes the role specification and a worked example, and identifies canonical topology families and the iteration-trajectory table.

C EC3 Role Discovery: Full Specification

D Six Canonical Topology Families

E Full SOPBench Iteration Trajectory

F Held-Out Test Evaluation

The held-out evaluation freezes the selected O3/I4 configuration and tests it on all 94 held-out tasks using the same oracle evaluator, constraint settings, and turn budget.

F Held-Out Test Evaluation

  • The 134 bank tasks are split into 40 validation and 94 test tasks using stratified sampling with random.seed(42); all test tasks expect successful outcomes.Source: F Held-Out Test Evaluation · p. 13
  • Search selects O3/I4 as the best configuration, achieving 70% on its 20-task validation batch with a six-agent ensemble topology.Table 8 reports the iteration trajectory using 20-task validation batches.Source: F Held-Out Test Evaluation · p. 13, Table 8 · p. 14
  • 62 of 94 held-out tasks pass all five oracle criteria, yielding a 65.96% test pass rate.Source: F Held-Out Test Evaluation · p. 14
  • The validation-test gap is 4 points: validation is 70% and test is 65.96%.The paper attributes the gap to higher variance in 20-task validation batches and selection of the peak validation iteration; it describes the 94-task test result as more stable.Source: F Held-Out Test Evaluation · p. 14

G Multi-Agent Turn Efficiency Analysis

Under SOPBench’s 20-turn budget, coordination overhead limits tool-call efficiency, although the ensemble still exceeds the published GPT-4o baseline. The analysis also characterizes topology diversity, document growth, and structural conditions for trace diagnosis.

G Multi-Agent Turn Efficiency Analysis

  • 26% turn efficiency at O3/I4 accompanies approximately 66% of tasks hitting the 20-turn limit, as routing, handoffs, and aggregation consume turns.The 26% figure means roughly one in four agent turns produces a tool call visible to the oracle evaluator.Source: G Multi-Agent Turn Efficiency Analysis · pp. 14–15
  • 70% validation pass rate and 65.96% test pass rate exceed the published GPT-4o baseline (58.96%) despite approximately 5 effective tool-calling turns out of 20.The fan-out pattern enables parallel specialist execution, while the aggregator synthesizes results that a single agent might miss through sequential processing.Source: G Multi-Agent Turn Efficiency Analysis · p. 15

H GED Matrix and Topology Gallery

  • GED distances exceed δstruct = 3, confirming that the repulsion mechanism enforces structural diversity.The GED matrix compares the three converged topologies.Source: H GED Matrix and Topology Gallery · p. 15
  • O1 centralizes routing and synthesis in BankManager, whereas O2/O3 separate fan-out routing from aggregation, with O3 using domain-specific role types.O3 assigns verification to CreditApplicationSpecialist and an Oracle stance to ForeignExchangeSpecialist for real-time data dependency.Source: H GED Matrix and Topology Gallery · p. 15

I SKILL.md Growth Statistics

  • K grows monotonically and is preserved across outer loops; R grows substantially during family transitions, T repopulates rapidly after clearing, and P remains relatively stable.Source: I SKILL.md Growth Statistics · p. 15

K Theoretical Grounding

Related papers

Loading 2603.22791v1…