PaperlayerView exact revision on arXiv

Source-linked AI summary

MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

Shengxuan Qiu, Haochen Huang, Shuzhang Zhong, Zizhuo Fu, Zebin Yang, Jianming Tong, Meng Li

arXiv:2610.40293v1cs.MA

TL;DR

Multi-agent collaboration changes the readiness and context of serving requests, but observed requests do not capture the rules that generated them. MAServeBench makes those rules configurable and records realized work for controlled comparisons. Its task-linked measurements connect collaboration patterns to parallelism, compute efficiency, KV pressure, and task completion.

  • Problem

    Request traces record realized work but not the collaboration rules that shape request readiness, batching opportunities, and memory demand.

    Source: §1 · p. 1

  • Method

    MAServeBench composes configurable collaboration stages for native workload generation and uses hierarchical Semantic Traces to preserve operations and dependencies for fixed-work release comparisons.

    Source: §1 · p. 2

  • Results

    MAServeBench measurements connect collaboration to serving parallelism, locality-related compute efficiency, and KV pressure affecting task completion.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The benchmark provides a reusable foundation for evaluating multi-agent serving and hardware across serving backends.

    Source: §8 · p. 10

Abstract

from arXiv · show

Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evolving context as agents coordinate and exchange results. This creates changing patterns of concurrency, computation, and memory demand that can expose different bottlenecks in LLM serving hardware. Yet benchmarking these effects requires capturing the collaboration rules that generate serving workloads, beyond the requests observed during execution. We present MAServeBench, a benchmark that treats multi-agent collaboration as a configurable dimension of serving workloads for LLM serving and hardware evaluation. Its composable stages capture how agents coordinate and exchange context in documented MAS workflows, enabling native executions to generate different workloads even when task inputs and tools remain unchanged. Hierarchical Semantic Traces preserve realized operations and dependencies for reproducible comparisons of legal release constraints at fixed work. Task-linked measurements connect these workloads to serving execution and GPU/NPU hardware behavior, revealing why serving efficiency does not always translate into timely task completion. The benchmark code is available at https://github.com/ShengxuanQiu/MAServeBench.

1 INTRODUCTION

Multi-agent collaboration changes when serving requests become ready and what context they consume, so request-level traces alone do not capture the rules that generate the workload. MAServeBench makes collaboration configurable and combines native workload generation, fixed-work traces, and task-linked hardware measurements.

1 INTRODUCTION

  • Dependent calls, synchronization, repeated exchanges, and changing contexts alter readiness, batching opportunities, and memory demand even for tasks with similar request counts and token lengths.Observed request traces record realized work, not the collaboration rules that generated it.Source: §1 · p. 1
  • MAServeBench generates collaboration-dependent workloads through reusable stages that vary coordination and context exchange independently of task bindings.The stages compose into workflows while respecting state dependencies.Source: §1 · p. 2
  • Hierarchical Semantic Traces preserve realized operations and typed dependencies for fixed-work comparisons of legal release constraints.These comparisons distinguish changing when fixed work is exposed from changing the collaboration rules that generate work.Source: §1 · p. 2
  • Task-linked measurements connect generated work and its exposure to serving batches, GPU/NPU costs, and complete-task SLOs.The benchmark also includes functional and replay checks to validate its measurements.Source: §1 · p. 2

2 MOTIVATION AND BENCHMARK REQUIREMENTS

Request-level workloads describe arrivals and token lengths but omit the collaboration rules that generate calls and contexts. MAS hardware evaluation therefore needs to represent work generation, control legal work exposure, preserve reproducible executions, and link task progress to hardware measurements.

2.1 Why Request-Level Workloads Are Insufficient

  • Arrival times and token lengths do not capture how collaboration choices change generated calls, contexts, and token work under shared task bindings.At fixed realized work, legal release changes must preserve artifact and state prerequisites; changing an artifact selection instead requires a new native realization.Source: §2.1 · p. 2
  • Hardware attribution requires linking task and stage identities to batches, operators, resources, and task-completion paths while separating backend choices from collaboration dependencies.This motivates measurements that distinguish execution opportunities from hardware costs and relate both to latency and deadline-compliant service.Source: §2.1 · p. 2, §2.2 · p. 3

2.2 Requirements for MAS Hardware Evaluation

  • The four requirements are representative workloads, controllable collaboration, reproducible execution, and hardware attribution.Together they require preserving workflow coordination and context delivery, varying legal release constraints at fixed work, retaining resolved inputs and dependencies, and connecting progress to serving and hardware measurements.Source: §2.2 · pp. 2–3

3 HIERARCHICAL COLLABORATION WORKLOAD ABSTRACTION

MAServeBench separates a logical workload’s collaboration rules from the realized operations and dependencies produced by a run. Its model specifies workflow composition, stage behavior, reusable templates, and a hierarchical representation of execution.

3 HIERARCHICAL COLLABORATION WORKLOAD ABSTRACTION

  • The abstraction separates a logical collaboration workload from its realized execution, which is recorded with resolved payloads, output contracts, and external delays.The Semantic Trace supports realization and replay of the operation graph.Source: §3 · p. 3, §3.4 · p. 3

3.1 Hierarchical Workload Model

  • A task model combines stage slots S, required dependencies and artifact interfaces D, workflow control Γ, and logical memory M.Stages have explicit input/output interfaces and local specifications; repeated activations create distinct stage instances and runtime sessions.Source: §3.1 · p. 3

3.2 Collaboration Stage Semantics

  • Each stage specifies participants, execution control, and context construction, while logical memory defines state scope and access rules.Control rules constrain operation readiness; context choices affect generated token work and opportunities for prefix/KV reuse.Source: §3.2 · p. 3

3.3 Reusable Collaboration Templates

  • Four reusable templates cover spawning, fork–join, refinement loops, and debate, and workflows compose them with atomic model, tool, or transformation stages.The templates share the participants, control, and context interface.Source: §3.3 · p. 3

3.4 From Logical Workloads to Realized Execution

  • A run realizes model, tool, transformation, and state operations connected by control, artifact, and versioned-state dependencies.Operations retain task, stage-instance, and session identity, linking serving and operator observations to task and stage progress.Source: §3.4 · p. 3

4 BENCHMARK DESIGN

MAServeBench combines configurable native workload generation, Semantic Trace replay, and target-side measurement to support controlled serving experiments. Its contracts distinguish collaboration rules and realized work from release constraints and backend configuration.

4 BENCHMARK DESIGN

  • MAServeBench combines native generation, hierarchical Semantic Traces, and target-side measurement for controlled serving experiments.Task bindings supply inputs, role instructions, tools, and evaluation criteria independently of collaboration rules.Source: §4 · p. 4, §4.1 · p. 4

4.1 Benchmark Suite and Workload Construction

  • Native realization executes collaboration rules and tools, while fixed-work replay preserves operations, resolved inputs, output contracts, and required dependencies.A change to artifact selection that alters resolved input produces a new native realization rather than a fixed-work replay.Source: §4.1 · p. 4

4.2 Semantic Trace

  • Versioned Semantic Traces record operations, typed dependencies, resolved inputs, generation settings, tool results, and measured durations for reproducible replay.Replay returns recorded tool results without external service calls, and downstream operations remain subject to the same dependencies.Source: §4.2 · p. 4
  • Trace, scenario, and system-configuration contracts separate fixed work and release rules from arrivals, measurement settings, SLOs, backend, hardware, and serving policies.Legal barriers may serialize independent operations only while preserving required dependencies and avoiding cycles.Source: §4.2 · p. 4

4.3 Execution and Hardware Measurement

  • Target-side measurements link task, stage, and operation identities to serving requests, scheduling, batches, operator shapes, and resources.Task metrics include root latency, completion-path timing, SLO attainment, and task goodput; application latency and native kernel/resource captures are measured separately.Source: §4.3 · p. 5

5 VALIDATING MASERVEBENCH

MAServeBench is validated against documented collaboration patterns, native task behavior, and recorded-work replay. The checks examine both whether realizations preserve serving-relevant semantics and whether fixed-work replay preserves operations, dependencies, and timing.

5.1 Source-Grounded Abstraction Validation

  • The validation suite exercises seven structures across 30 task bindings, producing 210 native realizations; twelve documented workflows cover all four local templates.Five workflows compose multiple templates, and a separate 21-instance fixed bank characterizes graph geometry and typed dependencies.Source: §5.1 · p. 5, Table 3 · p. 6
  • Across seven paired native comparisons, task-quality differences were 1.2–3.8 percentage points on success/accuracy and at most 0.019 on RACE.The comparisons covered six frameworks and four task sets with matched instances, model settings, and evaluation procedures.Source: §5.1 · p. 5

5.2 Preserving Serving-Relevant Semantics

  • Source mappings retain dependency order, artifact delivery, and context construction across research and coding workflows.Mappings also preserve research review ordering, isolated contexts, explicit compression, and versioned task/message prerequisites.Source: §5.2 · p. 5
  • Across three paired evaluations, request-input medians differed by at most 2.3%, while eight GPT Researcher bindings matched on 30 calls, depth 20, width 3, and peak in-flight 3.Three requests overlapped for 23% of root lifetime on average in both GPT Researcher arms; the largest input P90 shift was a 7.1% reduction for Magentic-One/WebArena.Source: §5.2 · p. 5
  • All 204 paired replay roots preserved resolved payloads, token counts, operations, and dependencies exactly, with no early predecessor release; root-latency differences ranged from −1.85% to +2.88%.Recorded-work checks also covered 1,032 completed requests and 186 required predecessor-release constraints within 1 ms.Source: §5.2 · pp. 5–6

6 EVALUATION

The evaluation combines native workload generation with fixed-work interventions and matched hardware probes. Across these settings, collaboration changes generated demand and execution shape, while release rules and KV capacity can alter complete-task latency and deadline attainment.

6.1 Experimental Setup

  • The evaluation uses native runs with 32, 64, or 128 users and paired fixed-work blocks that hold work and arrivals constant for release and resource interventions.Fixed-work serving evaluations use length-locked replay unless stated otherwise.Source: §6 · p. 6, §6.1 · p. 6

6.2 Generated Demand and Controlled Work Exposure

  • At 64 users, Spawn has 1.70× ordinary serving’s aggregate modeled arithmetic intensity and a 13.85× KV memory-time integral; Loop has comparable intensity but longer state residency.Spawn’s longer execution is 580 versus 109 seconds for ordinary serving.Source: §6.2 · p. 6, Figure 3 · p. 6
  • Native Selected delivery for HTTP reduces first-writer input by 7.5% yet increases total input by 28.4%, expanding calls from 25 to 30 and graph depth from 15 to 20.Editing the frozen Full graph cannot generate the additional revision and writing calls; a new native realization is required.Source: Figure 4 · p. 7, §6.2 · p. 7

6.3 Compute Demand

  • Weight locality reverses GPU projection rankings, while binary-verified SIMT Gather is 6.47–6.90× faster than SIMD on 950DT.For one-row 4096 × 4096 output projection, PRO 5000 is 1.78× faster with weight reuse, whereas A800 is 1.50× faster under eight-weight rotation.Source: §6.3 · pp. 7–8, Figure 6 · p. 8, Figure 7 · p. 8

6.4 Memory Demand

  • Expanding KV from 16 to 40 GiB reduces the processed/source prefill factor by 52% and root P95 by 17.5%.Native parallel branches produce concurrent KV residency and preemptions; greater capacity reduces preemptions and redundant prefill in every paired block.Source: §6.4 · p. 8, Figure 9 · p. 9

6.5 From Stage Execution to Task Latency

  • Interleaving fixed background traces raises target co-batching from 31.5% to 96.3% but changes overall root latency by only +0.743%; replacing two Loop backgrounds with Spawn and Fork–Join raises target mean latency 3.9–5.0%.Under interleaving, all 27 targets have shorter first-call but longer second-call scheduled residence; the second-call penalty dominates for GSM8K and PubMedQA, but not Science.Source: Figure 10 · p. 9, Figure 11 · p. 9, §6.5 · p. 9

6.6 From Hardware Efficiency to Task SLOs

  • At the second arrival rate, DAG achieves 100% versus Barrier’s 4% median deadline attainment under either KV budget; at the highest rate, extra KV raises output TPS 7.7% but deadline goodput remains near zero.The 40 GiB setting eliminates preemption observed at 16 GiB in two blocks per release rule, but the resulting latency savings do not bring enough tasks below the shared deadline.Source: §6.6 · pp. 9–10, Figure 12 · p. 10

6.7 Discussion

  • The discussion proposes reuse-aware compute, tiered KV memory, and programmable execution paths as hypotheses that must ultimately improve timely task completion.It also describes API-facing replay as a possible way to study replica routing and queueing while separating physical placement from logical collaboration.Source: §6.7 · p. 10

7 RELATED WORK

Related work spans multi-agent workflow evaluation, agentic serving benchmarks, and systems that exploit workflow or context structure. MAServeBench distinguishes itself by making collaboration semantics a workload-generation variable and linking realized workloads to serving, hardware, and task-SLO measurements.

7 RELATED WORK

  • Prior MAS workflow and coordination benchmarks evaluate agent outcomes, while MAServeBench examines how collaboration patterns change serving demand and task performance.The cited examples include AutoGen, MetaGPT, ChatDev, Magentic-One, AgentBench, and MultiAgentBench.Source: §7 · p. 10
  • Unlike trace profiling, graph-arrival replay, application characterization, or workflow-aware serving, MAServeBench varies collaboration semantics under shared task bindings and preserves each realization for controlled release and resource comparisons.Its measurements link these controls to serving, hardware, and task-SLO outcomes.Source: §7 · p. 10

8 CONCLUSION

MAServeBench provides a source-grounded, extensible benchmark that varies collaboration and context construction under shared task bindings. It preserves realized operations and dependencies for controlled serving comparisons and links those workloads to hardware behavior and complete-task outcomes.

8 CONCLUSION

  • Composable stages generate distinct native workloads from shared task bindings, while hierarchical Semantic Traces preserve operations and dependencies for fixed-work release and resource comparisons.Task-linked measurements connect collaboration to parallelism, compute locality, KV pressure, and task completion.Source: §8 · p. 10

A SOURCE WORKFLOWS AND FUNCTIONAL VALIDATION

A.1 Documented Workflow Mappings

  • The native demand population contains 210 realizations across 30 task bindings and seven structures, alongside a fixed bank recording operation counts, graph depth, level width, and typed edges.Each executable instance is connected to its template composition and input/output contract.Source: §A.1 · p. 14, Table A.1 · p. 14

A.2 Reference Workflows and Paired Functional Validation

Paired native realizations preserve source-workflow quality while retaining distinct execution demands, and the benchmark validates workload correspondence and replay conformance. Task-linked measurement contracts then connect operations to serving and hardware behavior.

A.2 Reference Workflows and Paired Functional Validation

  • The suite pairs six reference frameworks across four task sets, mapping worker orchestration to Spawn, research aggregation to Fork–Join, and revision to Refinement Loop.Source: §A.2 · p. 14
  • Quality differences are 1.2–3.8 percentage points on success/accuracy measures and at most 0.019 on RACE across paired source workflows and realizations.Source: §A.2 · p. 14, Table A.2 · p. 15

A.3 Source-to-Serving Workload Correspondence

B.1 Request and Dependency Checks

  • Across the model study, all 1,032 completed requests match recorded input/output counts, and all 186 predecessor edges meet release constraints within 1 ms.Source: §B.1 · p. 16

B.2 Concurrent and Cache-Enabled Replay

  • Concurrent and cache-enabled replay preserves canonical inputs, token counts, operation identities, and typed predecessors across all 204 root pairs, with no early predecessor releases.Source: §B.2 · pp. 16–17

B.3 Optional Exact-Token Replay

  • The CUDA Model Runner V2 adapter supports token-locked replay, while HTTP and Ascend adapters use length-locked replay; forced KV preemption and live cancellation require separate validation.Source: §B.3 · p. 17

C SYSTEMS, PROFILING, AND METRIC CONTRACTS

C.1 Arrival-rate and Task-SLO Contracts

  • Task goodput and SLO-constrained capacity are evaluated over tested loads that pass completion and backlog guards, with failed and incomplete tasks counted as attempts.Source: §C.1 · pp. 17–18

C.2 Legal Release, KV Capacity, and Shared Deadlines

The study compares legal release constraints and KV capacity under fixed work and shared deadlines, then tests how changing request limits affects task timeliness across workloads.

C.2 Legal Release, KV Capacity, and Shared Deadlines

  • At the second tested rate, DAG has higher median attainment and goodput than Barrier for all four deadlines and both KV budgets; across 96 paired contrasts, DAG is never lower.Source: §C.2 · p. 19
  • At 40/16 GiB, paired DAG ratios are 0.8451× root P95, 1.0774× output TPS, and 1.1884× mean decode batch; Barrier ratios are 0.9366×, 1.0324×, and 1.0582×.Source: §C.2 · p. 19

C.3 Request-Limit Sensitivity

  • At the highest ordinary-serving load, raising the request limit from 64 to 128 increases output TPS by 14.6% but reduces deadline-compliant task rate by 22.5%; Loop shifts the same way, while mixed-work median timely-task rate is approximately unchanged.Source: §C.3 · p. 19

D DEMAND AND FIXED-CALL CONTROLS

D.1 Legal Optional Control and Native Context Delivery

Legal release controls compare fixed realized work, while native context and collaboration choices generate different traces. The appendix also examines natural demand and fixed-call trajectories.

D.1 Legal Optional Control and Native Context Delivery

  • The fixed-work control preserves requests and required dependencies while comparing DAG release with a Barrier that adds optional ordering among independent analyses.Source: §D.1 · pp. 19, 21
  • For the four-call control, Arrival/DAG paired mean-latency ratios range from 0.997 to 1.003; Table D.1 reports legal-control effects under both context settings.Source: §D.1 · p. 21, Table D.1 · p. 22
  • Native execution regenerates artifacts and feedback decisions: the HTTP pair’s graph grows from 25 to 30 calls with an extra section revision, and from 15 to 20 calls with an extra writing pass.Source: §D.1 · p. 22, Table D.4 · p. 22

D.2 Natural Demand and Fixed-Call Trajectories

  • Natural-demand traces cover 30 bindings for each of seven structures; fixed-call comparisons preserve the resolved call set while varying dependency release across five workflows, three request limits, and three blocks.Source: §D.2 · p. 23, Figure D.4 · p. 23, Figure D.6 · p. 23

E COMPUTE GEOMETRY AND IMPLEMENTATION CHECKS

The section connects application workload geometry to controlled GPU/NPU measurements and traces captured matrix execution back to workflow stages.

GPU controls and batch exposure

Native workflow-to-matrix attribution

Application geometry coverage

Implementation checks

F PHYSICAL DATA SUPPLY, ATTENTION, AND STATE EVENTS

The appendix documents physical context and state measurements, then reports a fixed-work KV-budget comparison in which larger budgets reduce both prefill processing and root P95 latency.

F PHYSICAL DATA SUPPLY, ATTENTION, AND STATE EVENTS

F.1 Fixed-Work KV-Budget Intervention

  • 51.97% lower processed/source prefill factor and 17.49% lower root P95 at 40 versus 16 GiB; both decrease monotonically across budgets in every block.The comparison holds roots, arrivals, payloads, lengths, and dependencies fixed; all 576 attempted roots complete.Source: Figure F.2 · p. 26, §F.1 · p. 26, Table F.4 · p. 27

G.1 Balanced Fixed-Population Temporal Mixing

Fixed-work replay shows that temporal placement can change call residence and task latency, while composition and cohort comparisons expose different workload-demand effects. The appendix also contrasts matched and unmatched background demand and reports an earlier fixed-population mixing result.

G.1 Balanced Fixed-Population Temporal Mixing

  • All 27 targets have shorter first-call and longer second-call scheduled residence under interleaving; the second-call increase exceeds the first-call decrease for GSM8K and PubMedQA, but only partly offsets it for Science.The comparisons preserve the realized trace multiset and target arrivals while changing only the mapping of background traces to slots.Source: §G.1 · pp. 27–28, Table G.2 · p. 28

G.2 Background Composition and Demand

  • Scheduled residence rises by 12.705, 11.356, and 10.126 seconds across the three near-matched composition blocks, accounting for almost all target-latency changes.The composition perturbation increases background inputs by 4.45–4.48%, outputs by 4.78–4.82%, and calls by 5.45%.Source: §G.2 · p. 28, Figure G.1 · p. 28

G.3 Separate and Mixed Cohorts

  • Loop reaches 1.83× separate mean root latency, while Sequence falls to 0.53×; readiness-to-submission time accounts for 91.5% of Loop’s penalty and 92.3% of Sequence’s improvement.This separate-versus-mixed cohort comparison changes both co-execution and per-class load.Source: §G.3 · pp. 28–29, Figure G.2 · p. 29

G.4 Unmatched Background Demand

  • Heterogeneous backgrounds supply 4.92× as many input/output tokens and raise mean target latency to 1.77× the same-class value.The comparison keeps nine tagged Loop targets and 55 background roots per arm; all 27 targets per arm complete.Source: Figure G.5 · p. 30, §G.4 · p. 30, Table G.8 · p. 31

G.5 Earlier Fixed-Population Temporal Mixing

  • Earlier fixed-population temporal mixing yields a median target-latency change of −0.246%, with a range of [−0.796, +0.348]%, mainly in scheduled residence.The comparison keeps the entire population fixed and changes the mapping of background traces to arrival slots.Source: §G.5 · p. 31, Table G.9 · p. 31
Loading 2610.40293v1…