AgBench: Agentic AI Benchmarks for Personal AI Devices

1 Introduction

• Cloud-hosted agents transfer user inputs and execution context to cloud services, raising privacy concerns, while repeated model calls can incur substantial API costs.Source: §1 · p. 1
• Local execution can reduce cloud API costs and data exposure, but device resource constraints may lower task success and increase latency; existing benchmarks do not systematically characterize these trade-offs.Hybrid architectures may balance objectives, but their benefits remain unclear.Source: §1 · p. 1
• Local-only execution falls further behind cloud-only execution in task success at higher concurrency, while increasing concurrency can reduce task success for architectures using local agents.Local inference increases task completion time, and local-only execution eliminates cloud model API costs and sensitive-information exposure to cloud agents.Source: §1 · p. 1
• AgBench provides a benchmark suite, implementation, and execution traces for reproducible evaluation of agentic workloads on personal AI devices.The study systematically characterizes execution trade-offs and derives practical implications for deployment and performance optimization.Source: §1 · pp. 1–2

2 Background and Related Work

• On-device systems demonstrate the feasibility of local agent execution, while also identifying performance limitations on resource-constrained devices.Source: §2 · p. 2
• Hybrid systems partition planning and action execution or dynamically select device and cloud agents, but their benefits and limitations remain unclear.Source: §2 · p. 2
• Capability-oriented benchmarks cover realistic workloads and primarily measure task success, whereas systems-oriented benchmarks characterize execution performance, latency, costs, and bottlenecks.Source: §2 · p. 2
• Existing benchmark lines do not systematically explore trade-offs when agent execution moves from cloud to personal AI devices; AgBench combines task evaluation and execution measurements in one framework.Source: §2 · p. 2

3 AgBench

• AgBench evaluates agent execution using a methodology spanning workloads, configurations, agent execution, measurements, and a controlled experimental procedure.Source: §3 · p. 2

3.1 Benchmark Overview

• The benchmark runs 60 tasks across eight categories under controlled device, deployment-architecture, and concurrency combinations, capturing task outcomes and execution behavior.The overview describes a four-step workflow for constructing tasks, configuring runs, executing agents, and collecting measurements.Source: §3.1 · p. 2, Figure 1 · p. 3

3.2 Agent Workloads

• The workload suite contains 60 reproducible, programmatically verifiable tasks drawn from GAIA, TUA-Bench, and Terminal-Bench 2, spanning eight task categories.Each task includes an instruction, isolated environment, and task-specific verifier.Source: §3.2 · pp. 2–3

3.3 Execution Configuration

• The evaluation covers two devices, four deployment architectures, and concurrency levels of 1, 2, 4, and 8, using fixed local and cloud inference configurations.The devices represent discrete- and integrated-GPU designs with dedicated and unified memory, respectively; the complete matrix contains 32 configurations and about 162.07 million records.Source: §3.3 · pp. 3–4

3.4 Agent Execution

• Agents iteratively choose actions, invoke tools, and use returned results until completion or termination; hybrid agents can also delegate or consult through this loop.Tasks run in isolated Docker environments while concurrent executions share system resources and the local model service.Source: §3.4 · p. 4

3.5 Measurement

• AgBench measures task success, execution performance, cloud API cost, and cloud data exposure, supplemented by diagnostics of model and tool timing, interactions, usage, and resource utilization.Exposure counts distinct sensitive items observed in cloud inputs relative to sensitive items identified in initial task inputs.Source: §3.5 · p. 4, Table 3 · p. 4

3.6 Evaluation Method

• The study runs all 60 tasks across 32 configurations with a shared 7,200-second task limit and assesses stability using two additional runs at two concurrency levels on the RTX 5090.Across three runs, success counts vary by at most 6 out of 60 tasks, while overall task-success and goodput trends remain consistent.Source: §3.6 · p. 4

4.1 Task Success

• At concurrency 8, RTX 5090 LO falls from 35 to 12 successful tasks while CO changes from 50 to 48; on Max+ 395, LO drops from 31 to 22 while CO rises from 47 to 51.At concurrency 1, both hybrids improve on LO, but their advantage is inconsistent at higher concurrency.Source: §4.1 · p. 5, Figure 3 · p. 5
• At concurrency 1, LO matches CO in several task categories, but terminal-task success is lower; hybrids narrow that terminal-task gap on both devices.At concurrency 1, LO completes 9/12 local-file tasks and 7/10 retrieval tasks on both systems; on RTX 5090 it also matches CO in calculation (5/8) and system tasks (2/2).Source: §4.1 · p. 5, Figure 3 · p. 5
• At concurrency 8, RTX 5090 local runs encounter context errors, HCL can record these as verification failures, and repeated HCL errors can trigger over 100 delegations before two-hour timeouts.On Max+ 395, local-architecture failures instead include timeouts associated with slower inference.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 Execution Performance

• At concurrency 1 on RTX 5090, CO completes shared-success tasks in 2.07 minutes versus 7.01 for LO, 6.75 for HLL, and 15.80 for HCL.Cloud assistance does not necessarily shorten completion time when execution still depends on a local agent; local-model waiting dominates local and hybrid execution, while tools can dominate CO runtime.Source: §4.2 · pp. 5–6, Table 4 · p. 6, Figure 6 · p. 6
• On RTX 5090, LO goodput rises from 2.1 to 4.6 tasks/h as concurrency increases from 1 to 8, while CO rises from 12.1 to 19.3 tasks/h; only CO’s gains are roughly proportional to shorter suite runtime.For local-agent architectures, higher concurrency reduces successful completions and limits the benefit of shorter elapsed runtime.Source: §4.2 · p. 6, Figure 5 · p. 6, Table 5 · p. 6

4.3 Cloud API Cost

• Local-only execution eliminates cloud-model API charges, but hybrid architectures do not always cost less than CO because context, uncached-input and output charges, and coordination calls can offset savings.At concurrency 1 on RTX 5090, HLL’s higher uncached-input share and 30.2% more output tokens than CO make it more expensive overall; HCL calls rise from 770 to 3,206 between concurrency 1 and 8.Source: §4.3 · pp. 6–7, Figure 7 · p. 7, Table 6 · p. 7

4.4 Cloud Data Exposure

• LO exposes none of the 527 identified sensitive items to cloud agents, while CO exposes 76% across configurations; HLL exposure is lower than HCL’s, especially when the local agent works first.HLL exposure falls from 22% to 0% on RTX 5090 and from 13% to 2% on Max+ 395 between concurrency 1 and 8, a decline that may partly reflect lower task success.Source: §4.4 · p. 7, Table 7 · p. 7

4.5 Cross-Metric Trade-offs

• CO achieves higher task success and goodput than LO, while LO avoids cloud API cost and sensitive-information exposure; HLL can recover some success while remaining below CO on cost and exposure, but not matching its goodput.The preferred trade-off depends on whether fast, reliable completion or limiting cloud cost and exposure matters more.Source: §4.5 · p. 7

5 Design Implications

• Test local execution on the intended tasks and check whether task success and completion time meet requirements before deployment.Aggregate success can obscure category differences; if local execution meets requirements, it avoids cloud-model API costs and keeps task information off cloud models.Source: §5 · p. 8
• On the evaluated devices, increasing concurrency improves goodput but reduces task success, so applications prioritizing reliable completion should consider one task at a time.Use higher concurrency only after testing whether its goodput gain justifies the success drop on the target device and workload.Source: §5 · p. 8
• At low concurrency, HLL offers a potentially better balance than HCL: it has similar or slightly higher success, faster completion, and much less sensitive-information exposure.The local agent leads in HLL and asks the cloud for help when needed, unlike HCL, where the cloud directs local work and receives task files and progress reports.Source: §5 · p. 8

6 Benchmark Artifacts and Execution Traces

• The release includes the AgBench task suite, execution framework, traces, and system measurements.The suite provides task instructions, environment configurations, and verifiers; the framework implements four architectures and includes model-serving configurations and measurement-collection code.Source: §6 · p. 8
• The trace dataset records four architectures on two personal AI devices at concurrency levels 1, 2, 4, and 8, including interactions, timestamps, model usage, verification, and execution measurements.It totals approximately 115 GB uncompressed and is distributed as 9.0 GB of compressed archives.Source: §6 · p. 8

7 Conclusion

• AgBench covers 60 tasks across eight categories, comparing four deployment architectures on two personal AI devices under different concurrency levels.Devices complete a large subset locally, but broader task coverage and effective concurrent execution remain challenging.Source: §7 · p. 8
• Local success is near cloud success in some categories at low concurrency, but the overall gap widens at higher concurrency; local inference generally increases execution time.Fully local execution avoids cloud API charges and sensitive-information transfer, while hybrid execution can improve success without guaranteeing lower API cost than cloud-only inference.Source: §7 · p. 8
• Assess readiness against intended usage requirements, with deployment choices informed by task needs, resource-aware concurrency, and hybrid coordination adapted to execution progress.Local execution may suffice when it meets required task success and completion time; other scenarios may benefit from cloud assistance.Source: §7 · p. 8

A AgBench Task List

• AgBench comprises 60 tasks with resource profiles spanning CPU-heavy, memory-heavy, I/O-heavy, mixed, and low-footprint workloads.Profiles use median CPU time, peak memory, and physical read/write volume across eight C=1 executions.Source: Table 8 · p. 10, A AgBench Task List · p. 10

B Instrumentation and Recorded Data

• Execution records timestamp model interactions and tool operations and link events and agent messages to their task execution.Table 9 summarizes these records.Source: B Instrumentation and Recorded Data · p. 11, Table 9 · p. 11
• Model-use records capture local and cloud token usage, distinguish cached from uncached input, and support cloud-cost and local-service throughput estimates.Table 10 summarizes the recorded model-use information.Source: B Instrumentation and Recorded Data · p. 11, Table 10 · p. 11
• Resource records sample device- and container-level CPU and memory, container disk I/O, and device GPU utilization and memory usage.Table 11 summarizes these records.Source: B Instrumentation and Recorded Data · p. 11, Table 11 · p. 11

C Model and Agent Configuration

• Both devices serve Qwen3.8-27B locally with UD-Q6_K_L quantization and thinking enabled; the local model supports up to eight concurrent calls.Cloud inference uses DeepSeek V4 Flash with high reasoning effort, a 1,000,000-token context window, and a 384,000-token per-call output limit.Source: C Model and Agent Configuration · p. 11, Table 12 · p. 11
• Hybrid prompts specify how local and cloud agents use tools, exchange information, and continue tasks; HCL delegates execution stages, while HLL provides reasoning assistance to a Local Executor.The HCL and HLL instructions supplement the default Pi coding-agent prompt.Source: C Model and Agent Configuration · pp. 11–12

D Cloud API Pricing

• Table 13 lists the unit prices used to estimate cloud API costs.Source: Table 13 · p. 12

• Two additional 60-task-suite runs for each of LO, CO, HCL, and HLL at C=1 and C=8 provide three runs per configuration on RTX 5090.The repetitions use the main evaluation's settings and task ordering; the main analysis retains the original run.Source: E Repeatability of Key Results · p. 13
• Task success counts verifier-passing tasks out of 60, and goodput divides that count by the elapsed time of the complete run.Source: E Repeatability of Key Results · p. 13
• Repeated runs show consistent qualitative trends in task success and goodput, but minor numerical differences between architectures should be interpreted alongside run variation.Source: E Repeatability of Key Results · p. 13

• For HCL on RTX 5090, raising concurrency from C=1 to C=8 increases prefill throughput from 69.2 to 936.7 tokens/s, while decode throughput falls from 34.8 to 7.5 tokens/s.The uncached-input-to-generated-token ratio rises from approximately 2 to 125, consistent with context-error and re-delegation cycles; higher prefill throughput does not necessarily mean more effective execution.Source: F Additional Results · p. 13, Figure 11 · p. 14
• At C=1, LO and both hybrid architectures complete some tasks that CO does not, so aggregate success rates conceal differences in task coverage.Figure 9 compares successful-task overlap for architecture pairs at each device and concurrency level.Source: Figure 9 · p. 13, F Additional Results · p. 13
• On RTX 5090 at C=8, HLL's short run time and low task success are consistent with context-limit failures.Figure 10 tracks completed task executions over elapsed run time.Source: Figure 10 · p. 13, F Additional Results · p. 13
• Resource profiles suggest that moving inference from cloud to device can shift the dominant bottleneck from local tool execution to local model inference.The constraints differ by device: RTX 5090's smaller shared KV pool is associated with frequent context-limit errors, while slower inference on Max+ 395 coexists with frequent timeouts despite its larger KV pool.Source: F Additional Results · p. 14
• On RTX 5090 at C=8, unsuccessful HCL executions account for 85.2% of its cloud API cost; their $2.57 cost alone exceeds CO's total cost of $0.72.This supports the finding that hybrid execution does not guarantee lower cloud API cost.Source: F Additional Results · p. 14

• The task suite contains 527 distinct sensitive items across 15 tasks, including personal identifiers, credentials, personal records, and confidential business information.Repeated appearances of the same information within a task count as one item; Table 16 lists task-level counts.Source: G Sensitive Data Exposure Measurement · p. 14, Table 16 · p. 15
• For each configuration, exposure is based on sensitive items confirmed in recorded cloud inputs relative to each task's fixed sensitive-item set.The method defines the suite as 60 tasks and the exposed items as a subset of each task's sensitive-item set.Source: G Sensitive Data Exposure Measurement · p. 14
• Each item is counted once per task execution, and both successful and unsuccessful executions are included with a fixed denominator of 527.Source: G Sensitive Data Exposure Measurement · p. 14