MASEval: Extending Multi-Agent Evaluation from Models to Systems

Motivation

• Existing benchmarks fix the agentic setup and conflate model capability with framework implementation, leaving framework and coordination choices unanswered.Multi-agent evaluation must account for topology, communication protocols, memory, role differentiation, and orchestration logic.Source: §1 · p. 1

MASEval

• MASEval evaluates the complete system—agents, framework, and coordination logic—rather than the model alone.It is presented as a framework-agnostic evaluation library for multi-agent systems.Source: §1 · p. 2
• MASEval provides system-level infrastructure for comparing design decisions and framework implementations, treating both as relevant to effective multi-agent systems.Source: §1 · p. 2
• Its unified benchmark interface evaluates agents across multiple benchmarks with minimal integration overhead.Source: §1 · p. 2
• Multi-agent tracing records per-agent message histories for debugging coordination patterns, while the benchmark development toolkit supports creating new benchmarks without rebuilding evaluation boilerplate.Source: §1 · p. 2

Validation

• Across 3 benchmarks, 3 frameworks, and 3 models, framework choice impacts performance comparably to model choice, exposing a pattern hidden by model-centric benchmarks.MASEval reduces implementation effort by 83–91% for benchmark consumers and by 35–57% for benchmark producers.Source: §1 · p. 2

Multi-agent frameworks

• Existing frameworks span academic systems, developer tools, and provider SDKs, but differ in their supported architectures and implementation paradigms.Source: §2 · p. 2
• MARBLE restricts topologies and requires agents to use its own paradigm, preventing evaluation of custom frameworks.Source: §2 · p. 3

Evaluation tooling

• Observability tools focus on monitoring, while commercial platforms can introduce vendor lock-in rather than framework-agnostic benchmark execution.Source: §2 · p. 3
• None of the compared alternatives supports evaluating multi-agent systems across frameworks with per-agent tracing.This distinguishes MASEval from tools centered on monitoring, fixed paradigms, or single-agent evaluation.Source: §2 · p. 3

Design Principles

• MASEval evaluates complete agent systems, enabling comparison of architectural choices and framework implementations rather than model capabilities alone.Source: §3.1 · p. 3, §3 · p. 3
• Users control tools, agent behaviour, and metrics while MASEval supplies orchestration, tracing, and lifecycle management with minimal framework-specific assumptions.Source: §3.1 · p. 3
• Task definition, environment, agent logic, and evaluation are separated so each component can be varied independently to isolate its performance effects.Source: §3.1 · p. 3

Tracing and Extensibility

• Independent per-agent traces and lifecycle callbacks support debugging coordination failures, monitoring, experiment tracking, early stopping, and external integrations.Source: §3.1 · p. 3, §3.5 · p. 5

Design Tradeoffs

• The architecture trades turnkey platform features and lower initial setup for framework, service, and backend flexibility without dependency lock-in.Source: §3.1 · p. 3

Benchmark Lifecycle

• The five-phase lifecycle sets up registered components, executes agents with customizable orchestration, collects traces, evaluates full trajectories, and reports structured results across repeated tasks.Source: §3.4 · pp. 4–5

4.1 Experimental Setup

• The experiment covers 27 configurations spanning 3 frameworks, 3 models, and 3 benchmarks.Each benchmark includes two domains and all tasks within each selected domain.Source: §4.1 · p. 5
• The evaluation holds benchmark logic and experimental conditions constant while retaining each framework’s built-in prompts, tool mounting, and error handling.Judges, environment simulation, and attackers use Gemini-3.0-Flash across conditions.Source: §4.1 · p. 6

4.2 Results

• Framework choice produces performance differences comparable in magnitude to model choice within the same capability tier.Across six domains, mean cross-model and cross-framework ranges are 14.2 and 12.4 percentage points, respectively.Source: §4.2 · pp. 6–7, Table 2 · p. 6
• Framework effects vary by model: Haiku 4.5 scores 90.4 with smolagents but 59.5 with LlamaIndex on MACS Travel.The 30.9-point gap illustrates framework-model interaction rather than a universal framework ranking.Source: Table 2 · p. 6, §4.2 · p. 7

4.3 Implementation Effort

• MASEval reduces total benchmark implementation effort by 35–57%, with orchestration-layer interface code shrinking by 83–91%.The shared infrastructure supplies lifecycle management, trace collection, and execution infrastructure.Source: §4.3 · p. 7, Table 3 · p. 7

5 Conclusion

• MASEval combines multi-agent tracing, structured error attribution, and a unified benchmark interface for system-level evaluation.These capabilities support comparison of architectural choices and framework implementations.Source: §5 · p. 7
• Framework choice impacts performance comparably to model choice within a capability tier.The conclusion presents this result as a challenge to the model-centric evaluation status quo.Source: §5 · p. 7

Ethics Statement

• MASEval does not introduce new capabilities for language models.The infrastructure evaluates systems and benchmarks rather than extending model capabilities.Source: Ethics Statement · p. 8
• The project aims to lower barriers to systematic benchmarking and support reproducible comparison of safety failure modes and mitigation strategies.The authors state that this benefit outweighs the risk of accelerating more capable autonomous-agent development.Source: Ethics Statement · p. 8

• MASEval currently integrates four frameworks and seven benchmarks, while broader coverage depends on community contributions.Source: Broader Impact Statement · p. 8
• Scaling beyond five agents remains unexplored, although tracing overhead grows linearly and is negligible relative to agent execution costs.Source: Broader Impact Statement · p. 8
• Novel frameworks may require thicker adapters because memory, asynchronous execution, and tool-calling abstractions differ substantially.Source: Broader Impact Statement · p. 8
• MASEval prioritises flexibility over convenience, requiring more manual work and potentially slowing adoption outside research settings.Source: Broader Impact Statement · p. 8

• C. Emde conceived and implemented the library, built its infrastructure, documented it, ran experiments, analyzed results, and drafted the paper.Source: Contribution Statement · p. 8
• M. Gubri, S. J. Oh, and S. Yun provided weekly supervision and organizational and funding support.Source: Contribution Statement · p. 9

• Table 4 lists currently supported benchmarks and distinguishes single-agent from multi-agent settings.Source: Table 4 · p. 11
• Table 5 lists supported agentic frameworks integrated through thin adapters exposing execution, message history, tracing, and configuration capture.Source: Table 5 · p. 11

• Adding a framework requires implementing only _run_agent() and get_messages() through MASEval’s permissive adapter interface.Source: B Key Features · p. 11
• Per-agent message histories and component-specific traces capture coordination, model usage, tool activity, and simulator behavior.Source: B Key Features · p. 11
• The benchmark lifecycle exposes overridable hooks from environment and agent setup through execution and evaluation.Source: B Key Features · p. 11
• Two-stage evaluation separates trace filtering from metric computation, enabling reusable evaluators across different trace sources.Source: B Key Features · p. 11
• Custom execution loops, user simulation, parallel workers, callbacks, and configurable failure handling support varied evaluation workflows.Source: B Key Features · pp. 11–12
• Structured errors distinguish agent contract violations from environment and user failures, preventing infrastructure bugs from affecting scores.Source: B Key Features · pp. 11–12
• Thread-safe registries, model adapters, configuration snapshots, logging backends, adaptive queues, retries, timeouts, and task metadata support reproducible execution.Source: B Key Features · p. 12