PaperlayerView exact revision on arXiv

Source-linked AI summary

Planner-as-Router: Joint Plan-Time Model Routing for Cost-Efficient Multi-Agent Workflows

Vivek Kumar Singh, Preeti Priyam, Gautam Bhowmick

arXiv:2609.32917v1cs.AIcs.CLcs.MA

TL;DR

LLM-agent workflows can make frontier-model costs compound across chained specialist calls, while per-call routing cannot see dependency structure. PaR assigns tiers jointly during planning, and on EntBench it remains on the observed cost–accuracy frontier, with a preliminary compounding signal left as a hypothesis.

  • Problem

    Chained LLM-agent calls make frontier-model costs accumulate, while per-call routers cannot anticipate how an early error may affect downstream steps.

    Source: §I · p. 1

  • Method

    PaR assigns model tiers to subtasks as the planner builds the workflow, conditioning the joint assignment on its structure without a trained router.

    Source: §I · p. 1, §III · p. 2

  • Results

    On EntBench, PaR stays on the observed cost–accuracy frontier and cuts cost 44% against all-frontier routing for a 2.9-point accuracy drop.

    Source: §XI · p. 7

  • Takeaways & Limitations

    PaR offers a lighter-weight route to a comparable cost–accuracy result through the planner’s holistic view, without labeled data or a trained router.

    Source: §IX · p. 7

    The 54-task benchmark has wide confidence intervals (±6 points), and the planner uses one provider’s model and tiers, leaving mixed-provider settings untested.

    Source: §IX · p. 7

Abstract

from arXiv · show

Running large language model (LLM) agents in production gets expensive fast. A frontier model (the largest, most capable tier) is accurate but can cost 25 times what a small model costs per token, and the gap compounds once a workflow chains several calls together. Planner-as-Router (PaR) attacks this from a different angle. Instead of leaving model-tier selection to some component downstream, it folds the choice into planning itself. As the planner breaks a query into subtasks, it also assigns each one a model size tier (small, mid, or frontier, ordered by capability and price), so the dependencies between subtasks are visible before any specialist runs. Unlike per-call routers such as cascade routing, which look at one node at a time, PaR sees the whole workflow up front and needs no separate router model or training data. We evaluate PaR with EntBench, a benchmark of 54 enterprise agentic tasks across seven classes, graded by actually running the generated Structured Query Language (SQL) and MongoDB queries against live databases. Over 1,157 evaluations spanning eight routers and three seeds, PaR stays on the observed cost-accuracy frontier. It matches a sink-frontier heuristic (frontier model on terminal nodes only) in accuracy at comparable cost and a faithful FrugalGPT cascade at lower cost, and cuts cost 44% against all-frontier routing while giving up 2.9 points of accuracy. Several accuracy gaps fall inside the plus-or-minus six-point confidence interval of a 54-task study, so we frame PaR's advantage as frontier position rather than a clean accuracy win. We also report a preliminary observation, not a validated result: a small pilot hints that cheap routing may carry a hidden compounding penalty on compositional workflows, which we frame as a hypothesis for future measurement. PaR, EntBench, and all evaluation code are open source.

I. INTRODUCTION

PaR makes model-tier choice part of planning, addressing the blind spot of routers that decide one call at a time while workflow errors can propagate downstream. The paper presents PaR and EntBench, reports frontier-position results, and treats a possible compounding penalty as preliminary.

I. INTRODUCTION

  • Frontier-model calls can cost 5 to 25 times as much per token as small-model calls, and chained calls can make all-frontier workflows an order of magnitude costlier.This motivates deciding which steps need frontier capability and which can be served more cheaply.Source: §I · p. 1
  • A slightly wrong early answer can feed errors into every downstream node, a dependency risk that one-call-at-a-time routing cannot anticipate.FrugalGPT [1] is described as a cheap-first cascade that escalates when confidence drops.Source: §I · p. 1
  • Earlier routers operate per call, route a whole query, or assign tiers per agent; none assigns tiers jointly across a dependency-coupled workflow before execution.FrugalGPT [1], RouteLLM [2], Hybrid-LLM [3], and MasRouter [6] represent these different routing granularities.Source: §I · p. 1
  • PaR assigns a tier to each subtask as the planner writes the plan, using the whole workflow structure rather than routing calls in isolation.It needs no trained router or extra data and is implemented with a planner prompt plus a tier-aware dispatcher.Source: §I · p. 1
  • PaR stays on the observed cost–accuracy frontier, matching a sink-frontier heuristic at comparable cost and a faithful FrugalGPT cascade at lower cost.The paper also reports a 44% cost reduction against all-frontier routing for a 2.9-point accuracy drop.Source: §I · p. 1
  • The reported compounding-error signal is a preliminary observation, not a validated result, and is offered to motivate future measurement.The paper frames it as a hypothesis rather than an established finding.Source: §I · p. 2

II. RELATED WORK

  • EntBench fills a benchmark gap with execution-graded enterprise workflows, while existing benchmarks often omit per-node routing or cost measurements.BIRD [9] and Spider [10] use execution-based text-to-SQL grading but are single-step; AgentBench, τ-bench [11], and GAIA [12] do not cover live structured queries with cost breakdowns.Source: §II · p. 2

III. PLANNER-AS-ROUTER ARCHITECTURE

PaR treats tier assignment as a joint decision over the workflow DAG rather than a sequence of isolated node choices. A single planner pass approximates a cost-constrained objective because downstream success depends on upstream tier choices.

III. PLANNER-AS-ROUTER ARCHITECTURE

  • The routing objective chooses a tier for every workflow node to maximize cost-aware utility, with end-to-end accuracy depending on upstream tier choices.The workflow is a DAG whose nodes are specialist subtasks and whose edges indicate that one subtask consumes another’s output.Source: §III · p. 2
  • PaR emits all node tiers jointly in one planner forward pass, conditioning on the full decomposition instead of optimizing nodes in isolation.The paper claims empirical placement on the cost–accuracy frontier, not optimality.Source: §III · p. 2
  • Exact assignment is unavailable because the joint success distribution over the DAG is unknown at plan time, and the utility alone does not prescribe a graph-wide solution.The paper formulates the task as maximizing expected end-to-end accuracy subject to a cost budget.Source: §III · p. 2

A. Planner: Prompt Contract and Structured Output

The planner produces a structured plan with a tier and dependency list for each specialist subtask, plus a rationale for the choices. Its example assigns cheaper tiers to independent retrievals and a stronger tier to their reconciliation.

A. Planner: Prompt Contract and Structured Output

  • The planner is held fixed on Claude Haiku 4.5 across routers and must return a structured plan through a forced tool call.The tool call is pinned to produce_plan with a 1,000-token maximum; the system prompt is cached.Source: A. Planner: Prompt Contract and… · p. 2
  • Each planned subtask has an id, description, specialist, tier, and dependency list, while the plan requires one cost-rationale string.The rationale covers both the number of subtasks and each tier choice.Source: A. Planner: Prompt Contract and… · p. 2
  • The dispatcher executes subtasks in dependency order and escalates one tier once after failure.The plan assigns tiers before any specialist runs.Source: Figure 1 · p. 3
  • In the Cross-Recon example, two independent retrieval subtasks use the small tier, while their fan-in reconciliation uses mid.The reconciliation consumes both retrieval results, and the cost rationale records the tier choice.Source: A. Planner: Prompt Contract and… · pp. 2–3

B. Complexity Classifier

A lightweight classifier avoids a full planning call for queries labeled SIMPLE, emitting a small-tier single-subtask plan instead. COMPLEX queries proceed to the full planner, so PaR’s savings combine short-circuiting with tier reasoning.

B. Complexity Classifier

  • A lightweight Haiku classifier labels queries SIMPLE or COMPLEX; SIMPLE queries short-circuit planning into a single small-tier subtask.The specialist is inferred from the task class, and the full planner is not invoked.Source: B. Complexity Classifier · p. 3
  • COMPLEX queries continue to the full planner, and the cost rationale can be ablated while leaving the classifier unchanged.The ablation isolates the planner’s tier reasoning from classifier short-circuiting.Source: B. Complexity Classifier · p. 3
  • A Cross-Recon example assigns small tiers to independent MongoDB and SQL retrievals, then mid to the reconciliation depending on both.The plan’s rationale characterizes the retrievals as single-source reads and the final step as a fusion.Source: B. Complexity Classifier · p. 3

C. Dispatcher and Escalation

The dispatcher executes subtasks in dependency order at their planner-assigned tiers, with a fixed retry and one-tier escalation policy. Failed dependencies propagate nulls, and a per-task cost ceiling bounds spend.

C. Dispatcher and Escalation

  • Ready subtasks run in dependency order at their assigned tiers, with upstream outputs passed as context.Subtasks whose dependencies are satisfied form the ready set; independent subtasks could run concurrently, though this implementation runs them sequentially.Source: C. Dispatcher and Escalation · pp. 3–4
  • A failing subtask is retried at its assigned tier up to three times, then escalated exactly one tier and retried once.This fixed policy applies identically across routers; a failure after escalation is recorded as null.Source: C. Dispatcher and Escalation · pp. 3–4
  • A USD 75 kill switch caps per-task spend; reported cost counts specialist tokens but excludes the planner call as an identical additive offset across routers.Node costs use measured input, output, and cached token counts at each node's tier price.Source: C. Dispatcher and Escalation · p. 4
  • A null dependency causes its dependents to be recorded null without running, preventing downstream attempts from adding cost after upstream failure.The dispatcher isolates routing comparisons from cascade noise by not charging for dependent nodes that cannot run.Source: C. Dispatcher and Escalation · p. 4

IV. ENTBENCH

EntBench contains 54 tasks across seven classes, including single-tier capability probes and compositional tasks chaining specialists. It supports per-node measurement through execution-based grading.

IV. ENTBENCH

  • EntBench covers 54 tasks in seven classes, split between capability tasks probing a single tier and compositional tasks chaining several specialists.The task design allows success or failure to be attributed to specific tier assignments rather than only an end-to-end score.Source: §IV · p. 4
  • Generated SQL and MongoDB pipelines are executed against live databases, with correctness determined by matching the gold output rather than text similarity.SQL uses bag equivalence within tolerance; MongoDB is matched on aggregate output, while extraction, multi-tool plans, and policy actions have task-specific criteria.Source: §IV · p. 4
  • Execution-based grading catches bad joins, empty aggregates, and malformed plans that text comparison could accept as correct.The benchmark's task decomposition and execution grading enable per-node attribution of routing outcomes.Source: §IV · p. 4

V. EXPERIMENTAL SETUP

The evaluation holds planner and specialist components fixed to compare tier-assignment strategies, using live-database execution and a uniform retry policy. Design controls also include a faithful cascade baseline and a PaR-Lite comparison.

V. EXPERIMENTAL SETUP

  • Eight routers are compared with the planner and all specialist prompts held fixed, so only tier-assignment strategy varies; each primary router was planned to run 54 tasks across three random seeds.The seven primary routers had 1,134 planned runs, with one failed SourceFrontier run dropped; the PaR-no-rationale ablation ran eight Cross-Recon tasks across three seeds (24 runs), for 1,157 evaluations total.Source: §V · p. 4
  • The three tiers use Claude Haiku 4.5, Sonnet 4.6, and Opus 4.7, with measured PaR plans averaging 1.8 subtasks and assigning 45% small, 46% mid, and 8% frontier tiers.Monetary amounts are reported in USD; tier prices are specified per million input/output tokens.Source: §V · p. 4
  • Three independent repetitions and task-level paired-bootstrap confidence intervals are used because API calls are nondeterministic.The reported intervals use 10,000 resamples, with seeds averaged and task as the resampling unit.Source: §V · p. 4

VI. DESIGN RATIONALE

  • Generated SQL and MongoDB queries run against live databases instead of relying on an LLM judge, trading operational complexity for an execution-based verdict.The execution pipeline holds the planner, specialist prompts, retry policy, and seeds fixed across routers, varying only per-subtask tier assignment.Source: §VI · p. 5, Figure 2 · p. 5
  • Every router uses the same retry control—up to three assigned-tier retries, one escalation, then failure—so differences reflect tier assignment rather than retry effort.Caching and node-skipping were not adopted because per-specialist prompts were below the provider's cacheable minimum.Source: §VI · p. 5
  • The faithful FrugalGPT-style cascade uses a lightweight scorer to identify low-confidence cases for escalation and charges for the scorer.An earlier version failed to escalate because specialists emitted no usable confidence value.Source: §VI · p. 5
  • PaR-Lite matches the cheapest routers on accuracy but retains PaR-level cost, pointing to workflow-level planning rather than its classifier as the benefit source.PaR-Lite was designed as a control that skipped the planner in favor of a cheap classifier.Source: §VI · p. 5

VII. RESULTS

On the routing-sensitive tasks, PaR sits on the observed cost–accuracy frontier, with comparable accuracy and cost to SinkFrontier. It is significantly cheaper than AllFrontier and SourceFrontier, while class-level results vary by task.

VII. RESULTS

  • SQL-Compose is excluded from routing analysis because all routers score 0%, with 79% of its failures occurring at the first SQL-generation step.The paper attributes this to a specialist-quality bottleneck that tier assignment cannot fix.Source: §VII · p. 5, Table III · p. 6
  • PaR is significantly cheaper than AllFrontier (−0.0074 USD, CI [−0.0101, −0.0049]) and SourceFrontier (−0.0036 USD, CI [−0.0054, −0.0015]) with no significant accuracy cost.The reported cost and accuracy comparison holds on both task sets.Source: §VII · pp. 5–6
  • PaR sits on the Pareto frontier, matching SinkFrontier at comparable cost (26.1% versus 25.4% accuracy).At α = 1, PaR delivers about 1.6 times the accuracy-per-dollar of AllFrontier and 1.3 times that of SinkFrontier.Source: Figure 3 · p. 6, §VII · p. 6, Table III · p. 6
  • On SQL-Gen, routers converge near 50% accuracy and tier choice barely matters; on Mongo-Gen, frontier-heavy routers pull ahead modestly.PaR matches frontier-heavy routers on Extract, Policy-Action, and Cross-Recon while routing many subtasks to cheaper tiers; it ranks first in utility at each tested α.Source: §VII · p. 6

A. Planner Behavior in Practice

PaR’s planner assigns tiers according to task difficulty, routing single-step classes mostly to small models and compositional work to stronger tiers. A small pilot probes compounding errors, but its comparisons are preliminary and confounded.

A. Planner Behavior in Practice

  • Across 162 runs, PaR classified 43% of task-runs as SIMPLE and 57% as COMPLEX, averaging 1.8 subtasks per plan with 45% small, 46% mid, and 8% frontier assignments.SIMPLE tasks use a single small-tier subtask without a full planner call.Source: A. Planner Behavior in Practice · p. 6, Table V · p. 6
  • Single-step classes stay almost entirely small-tier, while compositional classes mostly escalate to mid and the hardest reconciliation and multi-tool subtasks reach frontier.The 43% simple-task short circuit accounts for cheap single-step classes; tier reasoning applies to the 57% complex tasks.Source: A. Planner Behavior in Practice · p. 6

VIII. A PRELIMINARY LOOK AT COMPOUNDING

  • The compounding-error factor ρ compares actual end-to-end failure with a naive independent-failure prediction; the figure illustrates how downstream corruption can amplify cheap-tier failures.The pilot covers 21 tasks and only two of eight routers, so this is a preliminary probe rather than a validated measurement.Source: §VIII · p. 6, Figure 4 · p. 7
  • The reported ρ values of 3.33 and 1.27 are illustrative, not a measured spectrum, because the rows use different task populations and the naive baseline may overpredict compositional accuracy.The baseline uses standalone capability-task accuracies to predict compositional end-to-end accuracy, so embedded-task difficulty may account for part of the apparent compounding.Source: §VIII · p. 7, Table VI · p. 7

A. Ablation: What the Design Choices Buy

Ablations suggest that both tier-rationale generation and integrated planner reasoning matter to PaR’s results. The cost-rationale probe is limited to one compositional class.

A. Ablation: What the Design Choices Buy

  • Removing the cost-rationale field reduced Cross-Recon accuracy to 0% across 24 task-seed evaluations at more than four times PaR’s cost.The targeted result was large and consistent across seeds, suggesting the rationale field steers tier assignment rather than being cosmetic.Source: A. Ablation: What the Design Choices Buy · p. 7
  • PaR-Lite matched the cheapest routers’ accuracy at 24.6% while costing USD 0.0105, indicating the benefit comes from integrated planner reasoning rather than a bolt-on classifier.Source: A. Ablation: What the Design Choices Buy · p. 7

IX. DISCUSSION AND THREATS TO VALIDITY

PaR reaches a strong observed cost–accuracy frontier without a trained router, while the paper bounds its conclusions by benchmark size and an unvalidated compounding-error pilot. The authors identify larger-scale evaluation and shared-task measurement as next steps.

IX. DISCUSSION AND THREATS TO VALIDITY

  • PaR’s holistic planning view reaches a good frontier point without a trained router or labeled data, and the approach can be added to graph-based frameworks as a prompt change plus dispatcher.The paper frames PaR as a lighter-weight route to a comparable result, not a head-to-head winner over learned routers.Source: §IX · p. 7
  • The 54-task benchmark yields ±6-point confidence intervals, so the paper emphasizes frontier position rather than separable accuracy gains.Absolute accuracies are 22–29%; the planner is one provider’s model assigning that provider’s tiers, and mixed-provider routing is untested.Source: §IX · p. 7

X. FUTURE WORK

  • Growing EntBench toward 300 tasks could tighten intervals, while a shared-task sweep across all 18 specialist-tier combinations could ground the pilot’s compounding measurements.The authors also identify mixed-provider routing as an untested confound to resolve.Source: §X · p. 7

XI. CONCLUSION

  • PaR matches a sink-frontier heuristic at comparable cost and a faithful FrugalGPT cascade at lower cost, while cutting cost 44% against all-frontier routing for a 2.9-point accuracy drop.The conclusion presents PaR as staying on the cost–accuracy frontier.Source: §XI · p. 7
  • The reported cheap-tier compounding-error signal is a hypothesis for future measurement, not a validated result; PaR, EntBench, and all code are open source.The proposed signal is that cheap-tier mixing may carry a propagation penalty that flat reporting would hide.Source: §XI · p. 8

Related papers

Loading 2609.32917v1…