Source-linked AI summary
EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures
Yuhai Long, Yuanxin Wei, Kai Wu, Jinhui Wei, Dan Huang, Jiangsu Du
TL;DR
Multi-agent LLM workloads are difficult to run efficiently on end-user devices because UMA decode is memory-bound and agent execution varies in drafting difficulty and tool stalls. EdgeAgent combines UMA-aware execution with agent-aware scheduling; its execution layer reaches up to 1.29× over Batch-SD, and full scheduling reaches 1.77× over optimized Batch-SD under extreme stalls.
Problem
Existing inference systems poorly handle concurrent, fragmented multi-agent workloads on end-user devices, where UMA decode is memory-bandwidth-bound and agents vary in drafting difficulty and tool stalls.
Source: §1 · p. 2, §2.5 · p. 4Method
Results
UMA-aware execution achieves up to 1.29× speedup over Batch-SD; under [1, 100]s stalls at N=4, EdgeAgent achieves 1.77× over optimized Batch-SD.
Source: §4.2 · p. 9Takeaways & Limitations
EdgeAgent’s cross-layer co-design improves hardware utilization under irregular multi-agent execution patterns on edge UMA devices.
Source: §6 · p. 13Graph compilers’ tensor-level dependency tracking flags disjoint CPU-GPU writes to a shared buffer as conflicts, forcing synchronization.
Source: §2.5 · p. 4
Abstract
from arXiv · showhide
Emerging multi-agent LLMs demand privacy-preserving edge deployment, yet current inference systems struggle with these collaborative workflows. Specifically, the memory-bound decode phase causes severe bus contention on unified memory architectures (UMA), paralyzing naive CPU-GPU co-execution. Furthermore, speculative decoding in multi-agent workloads faces extreme variance in drafting difficulty, alternating between complex reasoning and predictable structured generation. Compounded by frequent tool-induced stalls, this highly fragmented execution severely underutilizes hardware and defeats traditional static batching. We present EdgeAgent, a cross-layer inference system explicitly co-designed for edge UMA and multi-agent workloads. At the micro-architectural level, it bypasses rigid graph-compiler constraints to enable zero-copy UMA-aware tensor parallelism, utilizing asymmetric memory layouts to fully saturate both CPU and GPU compute units. At the scheduling level, it dynamically allocates draft budgets based on real-time sequence predictability to bound bandwidth waste. Concurrently, an asynchronous suspend-and-yield mechanism actively evicts stalled agents, ensuring continuous hardware saturation during unpredictable tool invocations. Extensive evaluations on an Apple M4 SoC demonstrate that the UMA-aware execution alone contributes a 1.29x speedup over batched speculative decoding. Adding the agent-aware scheduling lifts the full EdgeAgent system to a 1.77x speedup under extreme tool-use latencies.
1 Introduction
On-device inference matters as LLM use shifts toward collaborative agents with growing system access, but existing cloud and edge inference systems poorly fit these workloads. EdgeAgent combines UMA-aware execution with agent-aware scheduling to address memory contention, variable drafting difficulty, and tool-induced stalls.
1 Introduction
- Cloud-oriented systems target batched multi-user requests, while edge frameworks target single-request inference, leaving concurrent, fragmented agent workloads underutilized.Multi-agent tasks dynamically generate concurrent requests on one device.Source: §1 · p. 2
- Decode is memory-bandwidth-bound on UMA, limiting CPU-GPU tensor-parallel gains despite their effectiveness during compute-bound prefill.Speculative decoding amortizes target-model weight loading, but multi-agent workloads add further scheduling challenges.Source: §1 · p. 2
- Graph runtimes impede lock-free CPU-GPU tensor parallelism, static draft lengths waste bandwidth across varied tasks, and tool stalls leave idle compute unavailable to other agents.The bottlenecks include compiler synchronization and memory copies, rejected drafts, and failure to redistribute idle speculative budgets.Source: §1 · p. 2
- EdgeAgent co-designs UMA-aware execution and agent-aware scheduling, combining zero-copy tensor parallelism and SME kernels with dynamic draft budgets and suspend-and-yield.The system is evaluated using trace-driven, concurrent multi-agent workflows and is reported to improve throughput and task makespan over existing frameworks.Source: §1 · p. 2
2 Background and motivation
This section describes the memory-bound decode phase and the fragmented workloads created by multi-agent reasoning, structured generation, and tool stalls. It motivates task-aware speculative decoding and hardware and software changes for edge UMA.
2.1 LLM Inference Process
- Decode generates one token per step from the KV cache and is largely memory-bandwidth-bound, often dominating end-user inference latency.Prefill processes the prompt and builds the initial cache with greater parallelism.Source: §2.1 · pp. 2–3
2.2 Multi-Agent Inference Workloads
- Orchestrators reason over long contexts, while Workers generate structured code or tool instructions and may stall awaiting external tools.These differences create fragmented execution unlike the continuous, homogeneous processing assumed by traditional serving systems.Source: §2.2 · p. 3
2.3 Speculative Decoding
- On an Apple M4, speculative decoding peaks at 1.69× on the Structured Output Task at L=16, while longer drafts can degrade speedup through consecutive rejections.Drafting difficulty varies by task, so uniform draft lengths waste bandwidth on reasoning tasks and truncate predictable outputs.Source: §2.3 · pp. 3–4
2.4 The Edge UMA Memory Wall
- CPU–GPU co-run speedup falls from 1.36× over GPU on the 4096 × 4096 matrix to 0.99× over GPU on the 4096 × 128256 matrix.The larger case approaches the UMA bandwidth ceiling, with destructive bus arbitration limiting co-run performance.Source: Table 1 · p. 4, §2.4 · p. 4
2.5 Graph Compiler Limitations
- Graph compilers track dependencies at tensor granularity, treating disjoint CPU-GPU writes to a shared UMA buffer as conflicts and forcing synchronization.This can block heterogeneous execution despite writes targeting separate offsets of a contiguous memory block.Source: §2.5 · p. 4
3 EdgeAgent Design and Implementation
EdgeAgent couples multi-agent workflows with edge-UMA hardware constraints through collaborative execution and scheduling layers. The execution layer supports heterogeneous tensor parallelism, while the scheduler adapts draft budgets and yields slots during tool stalls.
3 EdgeAgent Design and Implementation
- EdgeAgent couples multi-agent workflows to edge-UMA constraints through execution and scheduling layers that coordinate resource allocation with hardware needs.The execution layer handles heterogeneous computation; the scheduling layer responds to agent predictability and tool-induced stalls.Source: §3 · pp. 4–5
3.1 UMA-Aware Execution Layer
- The UMA-aware layer enables zero-copy heterogeneous tensor parallelism with processor-specific weight layouts and graph barriers that preserve dependencies without materializing concat copies.CPU and GPU write disjoint output ranges in shared memory; a logical barrier orders downstream work while the heterogeneous fast path dispatches concurrent tasks.Source: §3 · p. 5, §3.1.1 · pp. 5–6
- SME micro-kernels use 16×64 tiling, pipelined loads, an 8MB L2 working set, and lock-free work stealing to address heterogeneous-core stragglers.The tiling overlaps memory loads with accumulation, while dynamic work stealing balances micro-tiles across faster P-cores and slower E-cores.Source: §3.1.2 · p. 6
3.2 Agent-Aware Scheduling Layer
- A profiled 16-slot budget is dynamically divided using Historical Accepted Length as a drafting-difficulty proxy, while reserving each agent minimum capacity.The scheduler uses accepted-token history to direct slots toward higher-yield agents; a minimum allocation preserves forward progress.Source: §3.2 · pp. 6–7, §3.2.1 · p. 7
- At verification boundaries, tool-stalled agents yield token slots to active streams, then resume from frozen KV caches without context re-prefill.Yielded slots can be redistributed to active agents; upon tool completion, the suspended agent returns to execution through in-place state restoration.Source: §3.2.2 · pp. 7–8
4 Evaluation
EdgeAgent is evaluated on trace-driven concurrent multi-agent workflows and Apple M4-family SoCs. Its UMA-aware execution, dynamic drafting budgets, and stall-aware yielding are assessed through end-to-end outcomes and component-specific tests.
4 Evaluation
- The system builds on MLX, restructuring its C++ graph executor for zero-copy UMA management and adding APIs to expose agent states to the scheduler.End-to-end evaluation measures makespan, concurrent throughput, and resilience to tool-induced stalls.Source: §4 · p. 8
4.1 Experimental Setup
- The evaluation uses an Apple M4 testbed, two 8B models, EAGLE-3 for speculative decoding, and synthesized reasoning, code, and tool traces.The M4 Pro is used for key ablations; the synthetic traces blend reasoning, code, and tool prompts at 1:2:2 and inject stochastic tool pauses.Source: §4.1 · p. 8
4.2 End-to-End Evaluation
- A 1.33× speedup over Batch-SD on mixed reasoning-and-coding workloads shows the benefit of dynamic budgeting; HAL gains grow as the mix shifts toward code and tool generation.The workload-composition sweep runs from 1:1:1 to 1:4:4, with arrival process and concurrency held fixed.Source: §4.2 · p. 9
- 1.77× speedup over optimized Batch-SD at N=4 under [1, 100]s tool stalls accompanies a makespan reduction from 213.1 to 120.6 seconds.Across evaluated configurations, UMA-aware execution alone reaches up to 1.29× speedup over Batch-SD; the full system combines it with HAL scheduling and stall-aware context switching.Source: §4.2 · p. 9, Figure 7 · p. 10
4.3 Evaluation on Tensor Parallelism
- At prompt length 1024, heterogeneous prefill reaches 304.5 tokens per second, 1.31× GPU-only throughput, and reduces TTFT for a 4096-token prompt from 21.0 to 16.2 seconds.For decode microbatches M∈{1, 4, 16}, zero-copy synchronization yields about 1.14× speedup; at M=128 it delivers 1.31× speedup by reducing synchronization overhead from +0.90 ms to +0.04 ms.Source: §4.3.1 · pp. 10–11, Figure 10 · p. 11, Figure 11 · p. 11
- On the down-projection at M=1, the custom SME kernel reaches 85.0 GFLOPS versus BNNS’s 19.0 GFLOPS, a 4.47× speedup.At M=4096, the SME kernel reaches 2.07 TFLOPS versus BNNS’s 639.3 GFLOPS, a 3.24× advantage.Source: Figure 12 · p. 11, §4.3.3 · p. 11
4.4 Dynamic Speculative Scheduling
- HAL-guided budget reallocation cuts execution time for 881 target tokens from 434 to 388 seconds, a 1.12× speedup over static allocation.The scheduler tracks each agent’s Historical Accepted Length using an exponential moving average and reallocates draft budgets toward higher-acceptance agents.Source: Figure 13 · p. 12, §4.4 · p. 12
4.5 Stall-Aware Context Switching
- Asynchronous yielding reduces three-agent execution time from 286 to 238 seconds and raises global throughput from 9.9 to 11.4 tokens per second.When an agent invokes an external tool, it yields its execution context so other agents can use the hardware during the stall.Source: §4.5 · p. 12
5 Related Work
Prior work addresses agentic serving, dynamic speculation, and heterogeneous edge execution through distinct strategies. EdgeAgent positions its approach around UMA-specific coordination and HAL-guided budgeting.
LLM Serving for Agentic Workloads
- Agent-serving systems such as SGLang and AgentInfer [24] co-design runtime memory with application semantics, while Continuum [20] and INFERCEPT address tool stalls on discrete GPUs.EdgeAgent instead uses UMA for in-place KV metadata freezing and zero-overhead context switching without moving physical bytes.Source: §5 · p. 12
Dynamic Speculative Scheduling
- Prior speculation methods include EAGLE-3 [21], Smurfs, AdaServe [22], EdgeLLM, Ghidorah, DualSpec [42], and AgentInfer [24]; EdgeAgent uses HAL to allocate budgets across concurrent streams.The cited methods respectively cover acceptance testing, SSM voting, cloud SLOs, draft-tree width, verification partitioning, semantic critics, and suffix automata.Source: §5 · p. 12
Heterogeneous Execution on Edge SoCs
- Earlier vision frameworks predict workload split ratios across heterogeneous processing units; Hello SME! and MPGEMM [13] optimize standalone GEMM micro-kernels, while EdgeAgent bridges the integration gap via a logical DAG topology barrier.The passage says execution frameworks rely on per-operator runtime heuristics without whole-graph analysis, while isolated libraries lack end-to-end integration.Source: §5 · p. 13
6 Conclusion
EdgeAgent couples agent-aware scheduling with hardware-level execution for concurrent multi-agent LLM workflows on edge UMA devices. It combines dynamic budgeting, stall-aware yielding, and zero-copy tensor parallelism to improve utilization under irregular execution.
6 Conclusion
- EdgeAgent tightly couples agent-aware scheduling and hardware-level execution for concurrent multi-agent LLM workflows on edge UMA devices.Its components include dynamic budgeting, stall-aware suspend-and-yield, and zero-copy architecture-aware tensor parallelism.Source: §6 · p. 13
- The combined design improves hardware utilization under highly irregular execution patterns.Source: §6 · p. 13
- The design argues for end-to-end co-design across semantic and physical layers in real-time agentic AI.Source: §6 · p. 13