PaperlayerView exact revision on arXiv

Source-linked AI summary

Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance

Abdelghny Orogat, Ana Rostam, Essam Mansour

arXiv:2602.03128v2cs.AI

TL;DR

Framework-level architectural effects remain poorly understood, and existing benchmarks lack standardized cross-framework comparison. The paper introduces a taxonomy and MAFBench, then evaluates nine frameworks under controlled architectural variation. It finds that architecture governs performance, with planning-format failures and topology mismatches among the reported effects.

  • Problem

    Prior work evaluates individual agent capabilities, but architectural dimensions and their effects lack characterization, unified cross-framework evaluation, and controlled evidence.

    Source: §1 · p. 2

  • Method

    The paper introduces a five-dimension architectural taxonomy and MAFBench, then conducts controlled comparisons across frameworks while holding the underlying LLM and task fixed.

    Source: §1 · pp. 1–2

  • Results

    Architectural design governs performance across nine frameworks; rigid planning schemas make up to 84.7% of questions unparseable.

    Source: §8 · p. 13

  • Takeaways & Limitations

    The study distills six evidence-based design principles to guide the construction and selection of multi-agent LLM frameworks.

    Source: §1 · p. 2

    Results are not verified across all model families, scales, or workloads, and Concordia alone does not characterize the GABM paradigm.

    Source: §4.3 · p. 11

Abstract

from arXiv · show

Multi-agent LLM frameworks are data-intensive systems that govern how agents orchestrate tasks, manage state, and coordinate decisions. These architectural choices control execution overhead, memory behavior, planning effectiveness, and coordination scalability. Their impact on system performance remains poorly understood. Existing benchmarks evaluate individual agent capabilities in isolation and lack standardized framework-level comparison. We make four contributions. We introduce an architectural taxonomy that decomposes multi-agent LLM frameworks along five dimensions: orchestration, memory, planning interfaces, specialization, and communication topology. We develop MAFBench, a unified evaluation suite that integrates existing benchmarks within a standardized execution pipeline. We conduct a controlled empirical study across nine frameworks, fixing the underlying LLM and varying only architectural design choices. We distill the results into six evidence-based design principles. Architectural design, not only model intelligence, governs performance. Orchestration alone increases latency by over 60x, and a minimal implementation of the same paradigm isolates that cost as implementation rather than paradigm. Schema-constrained planning interfaces reduce accuracy by up to 32 points through formatting failures, not reasoning errors. Communication topology drops coordination success from above 90% to below 30% under mismatched structure. Memory architecture controls recall and scalability independent of context window size, and no evaluated framework natively supports controlled knowledge revision.

1 Introduction

Multi-agent frameworks shape execution through architectural choices, but their system-level effects and controlled cross-framework comparisons remain poorly understood. The paper introduces a taxonomy and benchmark, then evaluates architecture while holding the underlying LLM fixed.

1 Introduction

  • The study evaluates the orchestration layer rather than applications built on top of frameworks, comparing five dimensions at fixed LLM and task.Reported performance ranges show order-of-magnitude variation in latency and throughput, alongside substantial variation in accuracy and task success.Source: §1 · p. 1, Table 1 · p. 1
  • A minimal GABM skeleton provides a second data point for separating paradigm costs from implementation costs while task, agents, and LLM stay fixed.The comparison also isolates the cost of environment mediation against other paradigms.Source: §1 · pp. 1–2
  • Framework-level architectural choices shape orchestration, storage, planning interfaces, and communication, yet their performance impact remains poorly understood.Prior work evaluates agent capabilities such as planning, memory, tool use, and collaboration rather than framework architecture.Source: §1 · pp. 1–2
  • The paper introduces a five-dimension taxonomy, MAFBench, a controlled study across nine frameworks, and six evidence-based design principles.The study fixes the underlying LLM and varies architectural choices; planning is evaluated across ten LLMs because interface effects interact with capability.Source: §1 · p. 2

2 Architectural Taxonomy

The taxonomy distinguishes general-purpose and task-specific multi-agent frameworks and places them in a framework-level architectural comparison. Its scope centers on general-purpose frameworks that expose architectural choices for controlled variation.

2 Architectural Taxonomy

  • A general-purpose framework comprises LLM-based agents, orchestration, communication, and a shared environment; a task-specific framework fixes an orchestration loop for one task family.Task-specific frameworks support extensions to tools, sub-agent definitions, planning modes, and hooks, but expose no shared environment.Source: §2 · p. 2
  • Coding assistants and personal-assistant systems instantiate task-specific frameworks, while the study focuses on general-purpose frameworks that expose orchestration, communication, and storage as design choices.This exposure permits controlled variation of one architectural dimension at a time.Source: §2 · p. 2
  • Table 3 organizes representative systems by architectural paradigm and design dimension to enable systematic framework-level comparison independent of models.The following section states the research questions guiding the evaluation.Source: §2 · p. 2

2.1 Architectural Paradigms

The paper distinguishes graph-based, role-based, and environment-mediated paradigms by who holds control-flow authority and how agents coordinate. A running trip-planning example illustrates their different execution structures.

2.1 Architectural Paradigms

  • Paradigms differ by which entity holds control-flow authority, and the taxonomy maps representative frameworks across architectural dimensions.The running example fixes a trip request and four roles; the paradigm implementations hold task, tools, and model constant.Source: §2.1 · pp. 2–3, Table 3 · p. 3

2.1.1 Graph-Based Frameworks.

  • Graph-based frameworks encode orchestration as a directed graph whose edges constrain control or data flow, with the developer holding control-flow authority.The LangGraph example advances state along authored edges and completes the task in four LLM calls.Source: §2.1.1 · p. 3

2.1.2 Role-Based Frameworks.

  • Role-based frameworks condition interaction on agent roles, with agents holding control-flow authority and delegating tasks at runtime.In the CrewAI example, specialists spend a second model turn consuming tool results, and internal delegation prompts are not retrievable.Source: §2.1.2 · pp. 3–4

2.1.3 GABM Frameworks.

  • GABM frameworks mediate coordination through shared environment state and a transition function, without exposing direct inter-agent communication.The deterministic skeleton completes the example in four calls; at equal call count it uses more input tokens than LangGraph because observations carry environment state.Source: §2.1.3 · p. 4

2.2 Architectural Design Decisions

The taxonomy separates agent behavior, storage, tool execution, multi-agent interaction, and environment representation into architectural design decisions. These dimensions are coupled, so controlled evaluation varies only choices that can change independently.

2.2.1 Single-Agent Characteristics.

  • Behavioral specification covers runtime roles, goals, and planning, while planning interfaces determine whether plans are elicited, their required format, and where they enter context.Graph-based frameworks encode behavior procedurally; CrewAI offers a plan-elicitation interface, while Concordia provides optional planning classes.Source: §2.2.1 · p. 4
  • Memory architecture describes how knowledge is represented, accessed, and updated across five memory types, with native support differing by paradigm.The available mechanisms are retrieval, context accumulation, or their hybrid; memory types requiring custom implementation are excluded.Source: §2.2.1 · p. 4
  • Tool execution varies between agent-bound tools, graph tool nodes, and environment-routed actions, while tool selection remains delegated to the LLM.Deterministic invocation can guarantee execution, whereas agent-bound invocation depends jointly on wording, agent loop, and model.Source: §2.2.1 · p. 4

2.2.2 Multi-Agent Characteristics.

  • Graph, role-based, and GABM paradigms differ in topology, communication pattern, and collaboration; those constraints shape information flow and coordination scalability.Graph frameworks use predefined edges, role-based frameworks use task-mediated delegation, and GABM frameworks route interactions through a central Game Master.Source: §2.2.2 · pp. 4–5

2.2.3 Environment.

  • Only GABM frameworks expose persistent shared environment state; other frameworks must re-derive shared context from message history.Orchestration and communication are coupled, so the evaluation varies topology in graph frameworks and storage in graph- and role-based frameworks rather than changing definitional dimensions.Source: §2.2.3 · p. 5, §2.2.4 · p. 5

2.3 Research Questions

The research questions examine how five architectural dimensions affect memory, planning, specialization, communication, and execution cost. They also ask whether costs or outcomes depend on the interface, topology, or implementation.

RQ1 (Memory Access Mode)

  • Which memory access modes govern multi-hop retrieval, long-range understanding, in-session learning, and controlled knowledge revision, and does hybrid retrieval–accumulation behave like either component?Knowledge revision is framed as a state update rather than an append.Source: §2.3 · p. 5

RQ2 (Planning Interface)

  • How does a planning interface’s output format affect accuracy and runtime, and do any effects arise from reasoning or parsing failures?Source: §2.3 · p. 5

RQ3–RQ5

  • The remaining questions test specialization mechanisms, topology-dependent coordination as agent counts grow, and orchestration costs on trivial tasks, including mediation’s paradigm versus implementation cost.Source: §2.3 · p. 5

3 MAFBench: A Unified Benchmark

MAFBench standardizes framework-level comparisons by combining existing benchmarks with a common execution pipeline. It evaluates memory, planning, specialization, tool use, and coordination under dimension-specific benchmark setups and scope constraints.

3 MAFBench: A Unified Benchmark

  • MAFBench wraps heterogeneous frameworks in a common pipeline with identical prompts and stopping criteria, semantic scoring, configuration parity, and cost-control routing.A minimal reference implementation helps separate paradigm cost from implementation idiosyncrasies when only one production framework exists.Source: §3 · p. 5

3.1 Memory Benchmarks

  • MemoryAgentBench evaluates four competencies—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—using standardized document ingestion and session initialization.The evaluation covers framework-native memory; an adapter interface also enables varying memory backends while holding the framework fixed.Source: §3.1 · pp. 5–6

3.2 Planning Interface Benchmarks

  • Three planning interfaces—NoPlan, schema-constrained Crew-Plan, and free-form Direct-LLM-Plan—are compared across GSM8K, CSQA, and MATH using the same plan-generating model.The benchmarks cover numerical, commonsense, and symbolic reasoning, while varying whether a plan is requested, its format, and where it enters context.Source: §3.2 · p. 6, Table 6 · p. 6

3.3 Specialization Benchmarks

  • Five CatDB datasets spanning regression and classification tasks compare four specialization strategies while holding task structure, data loading, model execution, and metrics fixed.The strategies vary identity framing, planning, procedural guidance, and data access; executed pipelines are scored rather than generated code.Source: §3.3 · p. 6, Table 7 · p. 6

3.4 Tool Use Benchmarks and Limitations

  • StableToolBench is integrated with its original queries, server, and scoring logic, but quantitative results are deferred because frameworks lack architectural control over tool selection.A pre-execution step bounds and equalizes tool subsets across runs; the benchmark spans domains and tool-interface complexities.Source: §3.4 · pp. 6–7, Figure 2 · p. 7

3.5 Coordination and Scaling Benchmarks

  • AGENTSNET tests five coordination tasks across three graph families and rewritten sequential, hierarchical, and centralized structures, sweeping tasks, graph families, and agent counts.The topology rewriting engine preserves agent sets while deriving structures from shared base graphs.Source: §3.5 · p. 7

4 Evaluation

The evaluation varies framework-level design choices under controlled conditions to examine their effects on execution cost and reasoning behavior. Across orchestration, memory, planning, and specialization, architectural choices produce distinct performance outcomes.

4 Evaluation

  • MAFBench varies one framework-level dimension at a time, mapping each to benchmarks, metrics, and research questions while holding other mechanisms fixed.Per-agent mechanisms are isolated before cross-agent coordination differences are evaluated.Source: §4 · p. 7, Table 8 · p. 7

4.1.1 Framework Overhead.

  • Concordia reaches 30.6 s (about 60× baseline), while its minimal GABM skeleton shows environment mediation costs milliseconds, locating the extreme overhead in implementation.Graph- and role-based frameworks add only modest overhead; the skeleton’s residual is three orders of magnitude below Concordia’s.Source: §4.1.1 · pp. 7–8, Figure 3 · p. 8

4.1.2 Memory Architecture Effects.

  • 44.9 for LangGraph (W=512) on AR exceeds pure retrieval (33.2) and Agno accumulation (19.1), while hybrid LangGraph leads TTL at 28.9.Retrieval-only LangGraph leads LRU at 30.4, whereas accumulation approaches that score only at large context windows.Source: §4.1.2 · p. 8, Table 10 · p. 9
  • Multi-hop forgetting remains below 5.0 across architectures, and accumulation grows steeply on high-query AR and TTL tasks; no native design supports controlled knowledge revision.Accumulation replays full histories for each query, while retrieval avoids repeated processing.Source: §4.1.2 · pp. 8–9, Figure 4 · p. 9

4.1.3 Planning Interface Effects.

  • Schema-constrained Crew-Plan reduces accuracy through formatting failures (84.7% on GSM8K for Qwen-7B) and raises runtime up to 31×; free-form planning avoids formatting failures.Planning-interface effects were evaluated across ten models because they interact with model capability.Source: §4.1.3 · p. 9, Table 12 · p. 10, Table 13 · p. 10

4.1.4 Agent Specialization.

  • Expert-guided procedure lifts Yelp and Volkert F1 above 94 versus label-based roughly 42 and 65; the profile tool helps data access (WiFi 37.3 to 78.6), not method.The profile tool supplies data access without prescribing a workflow.Source: §4.1.4 · p. 10, Table 14 · p. 10

4.2 Multi-Agent Evaluation

With the model, task, prompts, and protocol fixed, the evaluation varies communication topology and finds that coordination depends on the task’s information-flow needs. The reported findings are not verified across all model families, scales, or workloads.

4.2 Multi-Agent Evaluation

  • The controlled evaluation fixes the LLM, task semantics, prompts, stopping criteria, and metrics while varying communication topology.Agents communicate only with direct neighbors, isolating topology’s effect as agent populations grow.Source: §4.2 · p. 10, §4.2.1 · p. 11

4.2.1 Communication, Coordination, and Scaling.

  • Only fully connected communication consistently achieves consensus; every graph-based topology fails despite higher runtime and token expenditure.Consensus requires every agent to receive information from every other agent within bounded rounds, while sparse graphs can fragment into clusters.Source: Table 15 · p. 11, §4.2.1 · p. 11
  • Local tasks tolerate sparse structures, while topology dependence grows when tasks require global information.Coloring succeeds across graph families; for Consensus, only fully connected topology succeeds consistently, in three rounds independent of network size.Source: §4.2.1 · p. 11

4.3 Scope and Reflection

  • Results are not verified across all model families, scales, or workloads, although the paper reports architectural findings across its evaluated frameworks, datasets, and LLMs.Topology results use MAFBench’s rewriting engine rather than vendor frameworks, avoiding framework-implementation confounds.Source: §4.3 · pp. 11–12, §4.2.1 · p. 11

5 Multi-Agent Design Principles

The paper’s design principles prioritize matching architecture to task demands: execution structure, memory, planning, specialization, and communication all shape system behavior.

Principle 1

  • Orchestration execution structure governs latency and throughput, and two implementations of one paradigm differed by three orders of magnitude.A minimal implementation of the same paradigm avoids Concordia’s orders-of-magnitude latency, which the paper attributes to implementation cost rather than paradigm cost.Source: §5 · p. 12

Principle 2

  • Memory architecture should follow task semantics: retrieval supports factual recall, hybrid designs support in-session learning, and evaluated native architectures lack controlled revision.The paper advises against substituting larger context windows for memory architecture and identifies explicit editing mechanisms as missing for controlled knowledge revision.Source: §5 · p. 12

Principle 3

  • Planning interfaces should be permissive and plans optional; formatting failures, not reasoning errors, dominate accuracy loss for weaker models.Free-form planning removes formatting failures at modest runtime cost, while frontier models never violate the schema and still pay the extra call without accuracy gains.Source: §5 · p. 12

Principle 4

  • Specialization should encode expertise through procedures, because role labels do not activate domain reasoning and capability binding helps only when it supplies procedure.The reported findings use GPT-4o-mini across five CatDB datasets, and the authors note effects may differ for models with stronger instruction following.Source: §5 · p. 12

Principle 5

  • Communication topology should match information needs: sparse structures suit local coordination, while fully connected structures achieve consistent consensus.Full connectivity enables global agreement but incurs quadratic communication cost; sparse graphs with long paths fail even under large round and token budgets.Source: §5 · p. 12

Principle 6

  • Across the five dimensions, the authors conclude that framework execution semantics and interface design, rather than task difficulty or model capability, govern performance limits.They recommend selecting orchestration depth, memory access, planning format, and topology before tuning prompts or choosing models.Source: §5 · p. 12

6 Future Directions

The paper identifies extensions to MAFBench, memory lifecycle management, and safety measurement as directions for research on multi-agent systems.

Extending MAFBench

  • MAFBench currently covers five architectural dimensions under single-model configurations, with proposed extensions for cross-model sweeps, industry workloads, and controlled tool-use benchmarks.Repeated LLM-based evaluation at scale remains a practical barrier requiring cost-aware scheduling and selective replication.Source: §6 · p. 13

Memory Editing and Lifecycle Management

  • Every evaluated framework-native memory architecture is append-only, motivating explicit revision, removal, invalidation, and dependency tracking.The paper proposes versioned fact stores, reversible updates, and conflict resolution for concurrent agent writes.Source: §6 · p. 13

Multi-Agent Frameworks for Data-Intensive Applications

  • The authors frame automated construction of multi-agent architectures for data-intensive applications as an open systems problem connecting high-level task specifications to concrete configurations.The proposed compilation problem spans orchestration, memory, planning, specialization, and coordination.Source: §6 · p. 13

Safety and Security

  • Future benchmarks should measure prompt-injection propagation, memory-poisoning persistence, and unauthorized tool execution across configurations.The taxonomy identifies communication, storage, and orchestration as dimensions with direct safety implications.Source: §6 · p. 13

7 Related Work

Prior benchmarks evaluate model or agent capabilities while holding framework design fixed, and prior multi-agent studies do not isolate framework-level variables. MAFBench addresses this gap by fixing the LLM and task while varying architectural dimensions.

Prior benchmarks

  • HELM, Chatbot Arena, AgentBench, WebArena, and SWE-bench evaluate models or agents under standardized settings but keep execution frameworks or pipelines unchanged.These comparisons assess capability without enabling analysis of which architectural factors govern framework performance.Source: §7 · p. 13

Multi-agent architecture studies

  • Prior multi-agent architecture studies either vary architecture and LLM backbones together or evaluate one fixed design, leaving architectural effects uncontrolled.MAFBench fixes the underlying LLM and task while varying framework dimensions to attribute performance differences to design choices.Source: §7 · p. 13

8 Conclusion

Across nine frameworks, the paper concludes that architectural design governs performance alongside model intelligence, with distinct effects across orchestration, planning, coordination, and memory.

8 Conclusion

  • Across nine frameworks, architectural design—not model intelligence alone—governs performance.The taxonomy spans orchestration, memory, planning, specialization, and communication topology.Source: §8 · p. 13
  • Rigid planning schemas leave up to 84.7% of questions unparseable, making formatting failures rather than reasoning errors dominate accuracy loss.Source: §8 · p. 13
  • Only fully connected topology achieves consensus at scale, while memory architecture governs recall and scalability independent of context window size.Source: §8 · p. 13
  • Role labels activate no domain reasoning, but bound procedure lifts F1 above 94; a minimal environment-mediation implementation costs milliseconds.This isolates a production framework's order-of-magnitude overhead as implementation cost.Source: §8 · p. 13
  • No evaluated framework natively supports knowledge revision; the findings motivate research on adaptive architectures, principled coordination, and automated compilation.Source: §8 · p. 13
Loading 2602.03128v2…