Source-linked AI summary
PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents
Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye
TL;DR
LLM-agent benchmarks often conflate logical workflow planning with resource-constrained scheduling, leaving peak-load failures difficult to measure and attribute. PeakBench separates these capabilities and shows that strong planning does not reliably imply safe scheduling, while exposing resource information improves execution outcomes in model-dependent ways.
Problem
Current LLM-agent benchmarks often conflate dependency planning with resource-constrained scheduling, making workflow failures difficult to attribute.
Method
PeakBench evaluates executable multi-tool workflows through separate logical-planning and physical-scheduling protocols using validated dependencies, concurrency relations, and measured resource profiles.
Results
Across representative LLMs, strong planning does not reliably imply safe scheduling, while RASC provides measurable but model-dependent gains when resource information is exposed.
Takeaways & Limitations
PeakBench makes resource-aware orchestration a distinct, diagnosable capability rather than an automatic consequence of strong workflow planning.
Abstract
from arXiv · showhide
LLM agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.
Introduction
PeakBench addresses the gap between logical dependency planning and resource-constrained physical scheduling in multi-tool LLM-agent workflows. It evaluates these dimensions separately using executable workflows, execution-grounded annotations, and measured resource profiles to diagnose resource-blind execution.
- Motivation: Concurrent execution reduces sequential latency, but safe parallelization requires distinguishing prerequisite relations from independently runnable tool calls.Agents may hallucinate dependencies, causing unnecessary serialization, or miss prerequisites, causing execution-blocking errors.
- Problem: Resource-agnostic agents may greedily dispatch logically independent calls beyond finite capacity, creating the “peak load” failure mode.This conflation of logical independence with execution readiness can produce resource overflows and catastrophic service outages.
- Evaluation gap: Existing benchmarks mainly measure tool selection, argument generation, and end-to-end success under mostly serial execution, leaving parallelization and physical scheduling under-evaluated.A single task-success score obscures whether failures arise from planning, serialization, unsafe parallelism, or resource overload.
- PeakBench: PeakBench separates logical planning from physical scheduling with executable workflows, execution-grounded dependency and concurrency annotations, measured resource profiles, and finite-budget timestamp assignment.Dimension I tests recovery of prerequisites and concurrency; Dimension II tests scheduling with verified structure under finite resource budgets.
- Findings: PeakBench evaluations reveal that agents competent at logical planning can remain resource-blind during physical execution, while RASC tests whether resource metadata reduces avoidable overflow and improves utilization.RASC exposes resource information before scheduling as a simple resource-aware baseline.
Constrained Scheduling Problems
PeakBench formulates multi-tool execution as jointly satisfying data-flow prerequisites and finite-capacity resource constraints. Its two-dimensional evaluation separates logical dependency planning from physical scheduling to attribute failures and expose resource-blind bottlenecks.
- Overall Execution Objective: Multi-tool execution requires selecting invocations, preserving intermediate-data dependencies, and assigning activation timestamps while keeping concurrent resource demand within machine capacity.Logical success depends on sufficient invocations, arguments, and data dependencies; physical feasibility requires aggregate load LS(τ) to remain within capacity C.
- Decoupled Evaluation: PeakBench separates dependency-structure errors from scheduling errors by deriving an execution-grounded prerequisite relation R⋆ and evaluating logical success and resource feasibility independently.Dimension I compares predicted prerequisites with R⋆, while Dimension II fixes R⋆ and replays the schedule under measured resources and capacity.
- Logical Planning (Dependency Extraction): Stage 1 infers which workflow steps must be sequential and which pairs can validly run concurrently from the prerequisite relation R.An edge (vi, vj) indicates that vi must finish before vj begins because vj depends on vi’s output or side effect.
- Physical Scheduling (Resource Allocation): Stage 2 schedules invocations using durations, measured resource footprints, prerequisite constraints, and machine capacity, evaluating completion time, capacity breaches, and feasible-schedule utilization.The schedule must respect prerequisite completion times while keeping aggregate resource load within capacity.
- The “Resource-Blind” Bottleneck: A resource-blind scheduler can launch all ready steps as early as possible despite correct dependencies, causing capacity violations that PeakBench isolates in Dimension II.Such a scheduler observes prerequisite structure but not resource footprints ri or capacity C.
PeakBench
PeakBench benchmarks resource-aware execution of executable multi-tool workflows by separating dependency recovery from physical scheduling under finite resource constraints. Its workflows and resource profiles are empirically grounded, enabling evaluation of both logical structure prediction and safe, efficient orchestration.
- Benchmark design: PeakBench evaluates LLM agents as resource-aware workflow executors, not only functional tool selectors, using executable multi-tool workflows with empirically grounded dependencies and resource footprints.Each instance supports linked evaluations of execution-grounded dependency prediction and resource-constrained scheduling.
- Workflow synthesis: The benchmark contains 300 executable workflows stratified into 150 easy, 100 medium, and 50 hard tasks, with difficulty defined by workflow structure rather than linguistic complexity.Variation spans invocation scale, dependency depth, parallel branch width, and resource heterogeneity.
- Dependency annotation: PeakBench uses sandbox execution traces and perturbations to annotate which steps consume prior outputs and which can proceed independently, producing execution-grounded prerequisite structures.The resulting structure serves as ground truth for dependency recovery and as an oracle input for isolating scheduling ability.
- Dimension I: Logical Planning via Dependency Extraction: Dimension I evaluates dependency recovery with Graph Edit Distance and Edge F1, measuring structural transformations and the precision and recall of prerequisite edges.The DAG target is recovered from execution-grounded sandbox evidence rather than exhaustive human annotation.
- Dimension II: Physical Scheduling under Resource Constraints: Dimension II provides the verified prerequisite structure and historical resource profiles, then evaluates timestamp assignment that minimizes latency while preventing capacity breaches.Its dual-state cost model distinguishes persistent baseline overhead from transient execution surges, while CVA captures violation severity and strict MRU rewards utilization only for zero-CVA schedules.
Resource-Aware Scheduling Context (RASC)
Resource-Aware Scheduling Context (RASC) augments verified workflow scheduling with preexecution resource metadata and machine capacity, enabling agents to balance parallelism against aggregate resource limits. It changes only the scheduling context, requiring neither model retraining nor an external optimizer.
- Motivation: RASC gives agents the costs of ready tool calls and the shared capacity limits they must respect before execution.This supports safer scheduling when the workflow structure is already correct.
- Method: RASC takes the user question, verified workflow structure, machine capacity C, and each invocation’s estimated duration and measured resource footprint.Calls with satisfied dependencies may run concurrently, provided aggregate active-call load remains below capacity.
- Method: RASC changes the scheduling context at decision time without changing model parameters or the verified workflow structure.The resource profiles are exposed as additional scheduling information.
- Contribution: RASC shifts scheduling from launching every ready tool immediately to launching ready tools only when their combined resource pressure is acceptable.It preserves parallelism where capacity permits while discouraging resource bursts, without retraining or an external optimizer.
Experiment
PeakBench evaluates logical dependency recovery and resource-constrained scheduling separately across eight API models, controlled settings, and classical baselines. Results show that strong planning does not ensure safe scheduling, while resource profiles usually enable lower-latency, safer, and more targeted execution.
- Experiment: Eight frontier API models are evaluated under identical prompting and parsing protocols on PeakBench.The models are DeepSeek-V4-Flash, DeepSeek-V4-Pro, GLM-5, Kimi-K2.5, Claude Sonnet 4.6, GPT-4.1, GPT-5, and o3.
- Experiment: Dimension I tests dependency recovery, whereas Dimension II schedules verified workflows across small, medium, and large machine profiles.Dimension I uses GED and Edge F1; Dimension II uses Scheduling Latency, Capacity Violation Area, and strict mean resource utilization.
- Experiment: The study compares no-profile and RASC model settings with ASAP, serial execution, and resource-constrained list-scheduling baselines.No-profile models receive verified structure without resource telemetry, while RASC additionally receives structured resource profiles and machine capacities.
- Results: Strong dependency planning does not imply strong resource-aware scheduling, with GPT-5 and DeepSeek-V4-Pro illustrating the planning–scheduling gap.Across models, dependency-extraction success only weakly predicts whether the same workflow is scheduled safely under finite resources.
- Results: RASC usually lowers latency, reduces capacity violations, and improves strict resource utilization by replacing conservative delays with targeted staggering.ASAP is fastest but incurs many violations, serial execution has high latency, and DeepSeek-V4-Flash is the CVA exception under RASC.
Conclusion · A Additional Related Work · B Detailed Resource Characterization of Agentic Tools
PeakBench separates workflow-dependency recovery from resource-constrained scheduling, making peak-load failures measurable and attributable while showing that RASC reduces capacity violations and overload. It characterizes agentic tools using a five-dimensional resource footprint and instrumented cold- and warm-start profiling across representative inputs.
- Conclusion: PeakBench evaluates dependency recovery and tool scheduling separately under finite infrastructure constraints, making peak-load failures measurable and attributable.The benchmark targets agent-side workflow planning rather than accuracy-centric tool-use evaluation alone.
- Conclusion: RASC reduces capacity violations and overload, while models differ in how they convert resource profiles into efficient execution.
- A Additional Related Work: Prior agent work established tool use and increasingly evaluates multi-step workflows, API use, tool selection, and task completion across broad tool ecosystems.The related work includes ReAct, Toolformer, Voyager, SWE-bench, API-Bank, GAIA, BIG-bench, and newer MCP-oriented benchmarks.
- A Additional Related Work: Constraint- and cost-aware benchmarks examine explicit tool-use constraints, constrained preference optimization, and cost-optimal planning in dynamic environments.
- A Additional Related Work: Unlike infrastructure and workflow-choice systems that optimize after requests arrive, PeakBench tests whether agent workflows expose enough dependency and resource information to avoid peak load.
- B Detailed Resource Characterization of Agentic Tools: Each tool invocation is represented by a structured 5-dimensional footprint, r = ⟨ρcpu, ρmem, ρgpu, ρnet, ρio⟩.The dimensions cover CPU, memory, GPU, network, and disk I/O across the hardware stack used by LLM-driven agents.
- B Detailed Resource Characterization of Agentic Tools: The characterization tracks CPU and memory for general computation, GPU allocation for scarce acceleration resources, and network and disk I/O for environment-interactive tools.Examples include model-in-the-loop tasks, web scraping, external API communication, and database read/write operations.
- B Detailed Resource Characterization of Agentic Tools: For each eligible tool, three semantically different valid inputs are executed once in an instrumented sandbox, recording duration and five resource telemetry streams.Profiles explicitly distinguish cold-start persistent setup costs from warm-start transient invocation costs and store both static and dynamic components.
C Construction Details of PeakBench · D Simulated Machine Profiles · E RASC Scheduling Context Format
PeakBench combines controllable executable workflows with heterogeneous resource costs, simulated capacities grounded in observed tool loads, and structured scheduling contexts for dependency- and resource-aware execution. Its difficulty tiers scale servers and invocations, while RASC exposes dependencies, objectives, machine capacities, and per-step measurements.
- C Construction Details of PeakBench: The benchmark stratifies workflows into Easy, Medium, and Hard tiers with increasing server counts and tool invocations.Easy contains 1–2 servers and 2–4 invocations across 150 workflows; Medium contains 3–5 servers and 5–8 invocations across 100; Hard contains 6–10 servers and 10–15 invocations across 50.
- C Construction Details of PeakBench: PeakBench seeds task domains and tool-use patterns from existing API/MCP benchmarks, then synthesizes controllable, verifiable executable workflows.The benchmark is designed to stress both dependency reasoning and physical scheduling rather than reproduce the full distribution of real user requests.
- C Construction Details of PeakBench: Its tool catalog covers development, research, finance, content generation, and utility services, with tools assigned potentially overlapping CPU, memory, network, and disk-I/O costs.The multi-label resource taxonomy targets failures caused by overlapping heterogeneous costs.
- D Simulated Machine Profiles: Dimension II uses small, medium, and large machine profiles whose capacities are derived from observed single-step and earliest-layer loads.Small uses the maximum of the 95th-percentile single-step and 65th-percentile earliest-layer loads; medium uses step p98 and layer p80; large uses step p99 and layer p95.
- E RASC Scheduling Context Format: RASC provides a structured JSON-style scheduling context containing the question, steps, dependency structure, machine profile, and per-step resource measurements.The context includes fields for tool calls, prerequisites, earliest depth, machine capacities, and resource-cost metadata.
- E RASC Scheduling Context Format: The scheduling objective prioritizes obeying dependencies, staying within capacity, and minimizing safe makespan, with same-depth steps treated as concurrent.The context explicitly encodes primary, capacity, and latency constraints together with execution semantics.
- E RASC Scheduling Context Format: Per-step metadata records duration, average and peak CPU, peak memory, normalized costs, machine-capacity ratios, fit booleans, and resource classes.The required output contains per-step delays and a short reasoning summary.
F Additional Experimental Results · F.1 Capacity-Profile Sensitivity
Figure 5 shows that RASC’s gains vary by machine profile: they are largest on small and medium profiles, but smaller on large profiles. This pattern indicates that RASC primarily helps when dependency-valid schedules require physical resource coordination.
- F.1 Capacity-Profile Sensitivity: RASC delivers its largest gains on small machine profiles.Resource conflicts are more common on these profiles.
- F.1 Capacity-Profile Sensitivity: RASC also delivers large gains on medium machine profiles.Small and medium profiles are where resource conflicts are more common.
- F.1 Capacity-Profile Sensitivity: RASC’s gains are smaller on large machine profiles.Many workflows are already feasible on the large profile.
- F Additional Experimental Results: Resource conflicts are more common on small and medium profiles.This profile-dependent conflict pattern explains where RASC helps most.
- F Additional Experimental Results: Many workflows are already feasible on the large profile.Consequently, RASC’s gains are smaller there.
- F.1 Capacity-Profile Sensitivity: RASC mainly helps when dependency-valid schedules still require physical resource coordination.The machine-profile results support this interpretation.
F.2 Model-Level RASC Gains · F.3 Relationship between Logical Planning and Scheduling
RASC most strongly reduces CVA for several models, while slightly increasing it for DeepSeek-V4-Flash. Weak case-level correlations show that dependency-extraction success does not reliably predict safe and efficient physical scheduling.
- F.2 Model-Level RASC Gains: -1.457 CVA was the strongest reported RASC reduction, averaged over machine profiles, for GLM-5.The result is reported at the model level across machine profiles.
- F.2 Model-Level RASC Gains: -1.094 CVA was the RASC reduction for DeepSeek-V4-Pro, averaged over machine profiles.DeepSeek-V4-Pro was among the models with the strongest reductions.
- F.2 Model-Level RASC Gains: -1.092 CVA was the RASC reduction for Claude Sonnet 4.6, averaged over machine profiles.Claude Sonnet 4.6 was among the models with the strongest reductions.
- F.2 Model-Level RASC Gains: -1.053 CVA was the RASC reduction for o3, averaged over machine profiles.o3 was among the models with the strongest reductions.
- F.2 Model-Level RASC Gains: DeepSeek-V4-Flash showed a slight CVA increase under RASC.This contrasts with the reported CVA reductions for GLM-5, DeepSeek-V4-Pro, Claude Sonnet 4.6, and o3.
- F.3 Relationship between Logical Planning and Scheduling: Table 7 reports per-model, case-level correlations between Dimension I planning outcomes and Dimension II scheduling metrics for no-profile schedules.CVA and strict MRU are averaged over the three machine profiles for the same case.
- F.3 Relationship between Logical Planning and Scheduling: Weak correlations indicate that dependency-extraction success for a workflow does not reliably predict a safe and efficient physical schedule.The conclusion concerns a model’s success on dependency extraction for a specific workflow and its resulting scheduling quality.
F.4 Resource-Profile Noise Robustness
PeakBench evaluates robustness to noisy resource profiles, finding that RASC remains beneficial under 10–50% perturbations while its feasible-rate gain decreases as noise increases.
- Resource-Profile Noise Robustness: RASC remains beneficial under 10–50% resource-profile perturbations, but the feasible-rate gain decreases as profile noise increases.The analysis uses seven model runs with complete noise-robustness results.
- Resource-Profile Noise Robustness: Feasible rate serves as an auxiliary binary safety diagnostic, while CVA and strict MRU are the primary Dimension II metrics.The main text reports CVA and strict MRU as the primary metrics.
F.5 RASC Input Ablation
The RASC ablation shows that duration information alone does not improve feasibility, whereas exposing CPU and memory profiles enables scheduling gains by helping the model reason about capacity conflicts. CPU and memory explain nearly all observed improvement, while richer profile representations add only marginal benefit.
- Input ablation: Duration-only information does not improve feasibility over no-profile scheduling, while CPU and memory profiles produce improvement.The reported feasible rate is used as an auxiliary safety diagnostic for this ablation.
- Input ablation: The gains indicate that RASC helps the model reason about capacity conflicts rather than merely encouraging shorter or longer schedules.This interpretation follows the contrast between duration-only inputs and resource profiles.
- Input ablation: CPU and memory account for nearly all observed gains, whereas full profiles and machine-normalized ratios change results only marginally.The dominant bottleneck is the coarse capacity conflict captured by CPU and memory pressure.
- Input ablation: The main experiments retain the full structured RASC format for completeness while interpreting its effect primarily as resource-capacity awareness.The ablation motivates using the complete representation despite the marginal contribution of its additional components.