PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

Zichuan Xu, Yuyang Dong, Makoto Onizuka, Chuan Xiao

arXiv:2610.21098v1cs.CLcs.AIcs.LGcs.SE

TL;DR

Tool-agent scores cannot distinguish model failures from harness-limit failures, and prior benchmarks do not measure how much pressure effects coincide with those limits. ToolBottleneckBench varies one pressure on fixed tasks under frozen budgets and records failure endpoints; visible-menu pressure is the only model-facing pressure with a large penalty, which tracks completion-cap truncation.

  • Problem

    A tool-agent success score cannot distinguish model failure from harness-limit failure, and prior benchmarks do not measure how much load penalties coincide with harness limits.

    Source: §1 · p. 1

  • Method

    ToolBottleneckBench holds task families fixed, varies one of four pressures under frozen harness budgets, and records each failed episode’s first unmet obligation and terminal flag.

    Source: §1 · p. 1

  • Results

    The visible menu is the only model-facing pressure with a large penalty, which tracks completion-cap truncation; a sixteenfold cap reduces it from −43.7 to −2.7 pp in six truncation-heavy configurations.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Report pressure effects with their harness budgets and configuration truncation rates, since a tool agent’s success score reflects both its model and harness budgets.

    Source: §6 · p. 8, §7 · p. 8

    The findings use synthetic, exactly graded tasks and frozen harness budgets; the sixteenfold cap sweep tests one high cap on eight routed-menu configurations.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark that measures tool-agent performance under controlled pressures and fixed harness budgets. Synthetic enterprise tasks stay fixed while one pressure varies: the registered tool catalog, the visible tool menu, the certified workflow, or the encoding of returned records. An exact grader records the first unmet obligation of each failed episode and the harness limit, if any, that ended it. Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap. Raising the cap sixteenfold removes most of the penalty in the truncation-heavy configurations. Most failures end at a harness limit, and the failure record identifies the assigned pressure no more reliably than a count of tool calls. The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch. We recommend reporting pressure effects with the harness budgets under which they were measured. We release the benchmark at https://github.com/zichuanxu/tool-bottleneck-bench.

1 Introduction

ToolBottleneckBench addresses the difficulty of separating model failures from harness limits by varying one pressure at a time on fixed tasks under frozen budgets. Its results show that visible-menu pressure is the only model-facing pressure with a large penalty, which tracks completion-cap truncation.

Motivation

  • A success score or first-failure label alone cannot distinguish model failure from harness limits, and prior benchmarks do not measure how much load penalties coincide with those limits.A large menu can truncate an episode before any tool is chosen, although the grader records a tool-selection failure.Source: §1 · p. 1

Benchmark design

  • The benchmark fixes each task family, varies one of four pressures, and records the first unmet obligation together with the harness limit that ended each failure.The pressures are the registered tool catalog, visible tool menu, certified workflow, and returned-record encoding.Source: §1 · p. 1

Research questions

  • The study asks which pressures reduce verified success, how losses coincide with harness limits, whether failure records reveal assigned pressure, and which repairs outperform configuration switches.The overview specifies matched variants and frozen budgets for the main results.Source: §1 · p. 1, Figure 1 · p. 2

Main findings

  • Only visible-menu pressure has a large model-facing penalty; most failures end at harness limits, and only two repairs that do not improve model behavior beat every configuration switch.Under the routed menu, the penalty tracks completion-cap truncation across configurations (Pearson r = −0.95); 91.5% of graded failures end at a harness limit, while 5.5% are wrong conclusions.Source: §1 · p. 2

Contributions

  • The authors release the benchmark generator, grader, scored rows, and analysis code, alongside a repair protocol that tests repairs with and without pressure and against configuration switches.The protocol is part of the stated contribution, not a claim that every repair succeeds.Source: §1 · p. 2

2 Related Work

Prior tool-use research includes API-call benchmarks, realistic environments, controlled load studies, harness comparisons, and failure diagnosis. ToolBottleneckBench combines controlled pressure assignment with recording harness limits and comparing repairs against configuration switches.

Tool-use benchmarks

  • Realistic environments and live-server benchmarks often change catalog, workflow, and observation format together, making score differences hard to attribute to one load.Earlier benchmarks score correct API calls; StableToolBench stabilizes unstable APIs (Guo et al., 2024).Source: §2 · p. 2

Controlled studies of tool load

  • Controlled studies examine individual loads, while ToolBottleneckBench varies catalog, menu, workflow, and encoding on matched task families and records the ending harness limit.ToolMenuBench separates gold-tool visibility from tool selection (Babu and Iyer, 2026b,a), a distinction this benchmark follows.Source: §2 · p. 2

Harness effects

  • Harness studies report model-and-harness results and show that harness changes can reorder outcomes, but this work measures controlled pressure effects alongside specific harness limits.Prior work also studies interface and serving components, context growth, memory, and deliberation (Yang et al., 2024; Wang, 2026b; Lukassen et al., 2026; Zeng et al., 2026; Lodha et al., 2026; Chen et al., 2026a; Wang, 2026a; Cuadron et al., 2025).Source: §2 · pp. 2–3

Failure diagnosis and repair

  • Prior failure research localizes errors, patches harness flaws, and tests repairs; ToolBottleneckBench combines assigned pressures and wrong-layer controls, then compares each repair with configuration switches.Exact graders also need validity audits (Bhat et al., 2026; Gurram, 2026; Zhao et al., 2026).Source: §2 · p. 3

3 ToolBottleneckBench

ToolBottleneckBench uses matched variants of fixed task families to measure the effects of assigned pressures under frozen harness budgets. An exact grader checks ordered success obligations and records how each failed episode ended.

3 ToolBottleneckBench

  • A controlled bottleneck placed on a task is an assigned pressure; the benchmark measures its performance effect and checks whether the failure record points to it.Source: §3 · p. 3

3.1 Task Families and Matched Variants

  • Each task family fixes the request, decision facts, evidence rule, permitted final state, and operational tools; a deterministic compiler creates variants differing in one pressure.Matched variants keep the correct decision and facts constant, so paired success differences measure the changed pressure.Source: §3.1 · p. 3

3.2 Four Assigned Pressures

  • The four pressures change the registered catalog, visible menu, certified workflow, or returned-state encoding while preserving the task family.Catalog success uses intention-to-treat scoring when required tools are not retrieved; added menu tools are route neighbors; workflow levels increase required reads and context; encodings preserve the same record values.Source: §3.2 · pp. 3–4

3.3 Harness Contract

  • A frozen harness contract sets completion, episode, call, and turn limits; the main capacity runs use an oracle-assisted routed menu that narrows after the first operational-tool call.The completion cap is 1,024 tokens per request, and episodes can stop early when success becomes impossible within the call limit.Source: §3.3 · p. 4

3.4 Verified Task Success and Failure Record

  • Verified task success is the conjunction of nine ordered obligations, from tool availability and selection through completion integrity.The exact grader accepts any dependency-valid call order and uses no model judge.Source: §3.4 · p. 4
  • Each failed episode’s failure record pairs its first unmet obligation with a terminal flag naming how it ended, such as truncation, a budget limit, an early stop, or a wrong conclusion.The terminal flags have a defined precedence, and the benchmark audits its generator and grader.Source: §3.4 · pp. 4–5

4 Experimental Setup

The experiments evaluate served model-and-route configurations using paired contrasts, then test pressure-targeted repairs against other repairs and full configuration switches. Repair comparisons are re-executed on the same stressed cells.

4.1 Configurations and Data

  • The evaluation unit is a served model, provider route, and harness configuration; 21 configurations from seven organizations cover 26,576 scored conditions.Of these, 20,408 are graded by exact verified task success; some analyses use 20 configurations because one model lacks specified rows.Source: §4.1 · p. 5

4.2 Contrasts and Statistical Analysis

  • Capacity contrasts are paired within task families, with cluster-bootstrap intervals over 36 families and Holm correction for each set of eight p values.Sensitivity analyses resample intents and configurations; discovery contrasts resample 18 required-tool clusters.Source: §4.2 · p. 5

4.3 Repairs and the Switch Comparison

  • The study tests pressure-specific routers, prompts, and interface guides, plus a controller that issues calls itself, using stress gain, selectivity, and wrong-layer advantage to assess repairs.Repairs and complete configuration switches are re-executed on the same stressed cells; superiority is assessed against every switch using paired 95% intervals.Source: §4.3 · p. 5

5 Results

Across the tested pressures, only a larger visible tool menu has a large, robust penalty on verified task success; much of that penalty coincides with completion-cap truncation. Most failures end at harness limits, and only two repairs outperform every configuration switch on success.

5.2 How Much of the Menu Penalty Occurs with the Completion Cap?

  • The six truncation-heavy configurations’ pooled menu contrast shrinks from −43.7 to −2.7 pp [−4.7, −0.9] when the completion cap rises sixteenfold.At V = 128, truncation falls from 30.6–69.8% at the low cap to at most 1.4% at the high cap; the mild- and zero-truncation controls move by 2.8 and 1.0 pp.Source: Figure 2 · p. 6, §5.2 · pp. 6–7

5.1 Which Pressures Reduce Verified Task Success?

  • Growing the visible menu from 8 to 128 schemas changes VTS by −12.3 pp [−14.6, −9.9], while workflow and returned-state encoding effects are small or not detectable.With the full menu visible, the contrast is −17.5 pp [−20.2, −14.9]; 34 of 36 families degrade. The workflow effect is +0.8 pp [−1.2, 2.8], and compact encoding changes VTS by −3.7 pp [−5.7, −1.9] but does not survive correction.Source: §5.1 · p. 6, Figure 2 · p. 6

5.3 Where Do Failures End, and What Does the Record Reveal?

  • Harness limits end 91.5% of the 3,725 failures; all 849 failures labeled “selection” made no call, including 838 cut off at the completion cap.The failure record predicts assigned pressure at 48.9% balanced accuracy, below binned call counts alone at 54.8%; richer encodings reach 51.3%–54.1%.Source: §5.3 · p. 7
  • At 30 certified calls, VTS is 0.0%; 69 of 80 episodes exceed the episode budget and 9 are stopped as infeasible.No episode executes more than 23 calls. At catalog size R = 2,048, required tools appear in 19.4% of flat-retriever shortlists versus 77.8% of connector-first-router shortlists.Source: §5.3 · p. 7

5.4 Which Repairs Help, and Do They Beat Switching?

  • Three repairs activate by changing which tools reach the model or who issues calls, while neither prompt repair activates.The public-handoff controller raises structural-holdout VTS by 96.6 pp [94.6, 98.5] while taking execution away from the model; both prompt repairs have no detectable activation.Source: §5.4 · p. 7
  • Only the router and controller beat every configuration switch on success, despite neither improving model behavior.The router is success-superior for all 21 incumbents; the controller is success-superior for all 17 incumbents and cost-superior for 4.Source: §5.4 · p. 7

6 Discussion

Success scores should be read alongside harness budgets: the visible-menu penalty occurs mostly where configurations run out of their completion cap. Failure labels locate failures but do not explain them, and the authors recommend reporting pressure effects with budgets and truncation rates.

6 Discussion

  • In six truncation-heavy configurations, raising the completion cap sixteenfold reduces the menu penalty from −43.7 to −2.7 pp, while a small penalty remains.A larger menu drives more episodes into the cap, almost always before the first call; under a large menu, success rankings partly reflect how configurations spend the cap.Source: §6 · pp. 7–8
  • Most failures end at harness limits, and every “selection” failure comes from an episode that made no call.The failure record identifies assigned pressure no better than a count of tool calls; the limit that fired says how the episode ended, not why the agent failed.Source: §6 · p. 8, Figure 3 · p. 8
  • Repairs that activate change which tools reach the model or who issues the calls, but only the router and controller beat every configuration switch on success.The router does so by construction and the controller by delegation.Source: §6 · p. 8, Figure 4 · p. 8
  • Report pressure effects with harness budgets and each configuration’s truncation rate, pair first-failure labels with the fired limit, and compare repairs against configuration switches on the same cells.Source: §6 · p. 8

7 Conclusion

ToolBottleneckBench shows that tool-agent success reflects both model performance and harness budgets. Its findings are measured under fixed budgets, with future work proposed on episode budgets, call limits, and live tool servers.

7 Conclusion

  • The visible menu is the only model-facing pressure with a large penalty, and a sixteenfold completion-cap increase removes most of it in truncation-heavy configurations.Most failures end at harness limits, while only two repairs beat every configuration switch and neither improves model behavior.Source: §7 · p. 8
  • ToolBottleneckBench varies one pressure at a time on fixed tasks under fixed harness budgets and records which limit ends each failed episode.Source: §7 · p. 8
  • Future work can vary the episode budget and call limit, as the study varied the completion cap, and test live tool servers.Source: §7 · p. 9

Limitations

The findings are bounded by the frozen harness budgets and by the study’s synthetic task environment, exact grading, and fixed prompts. Configurations pair models with provider routes, and public tasks may have been seen by models trained after release.

Limitations

  • The findings hold at frozen harness budgets; only the completion cap varies, and its sixteenfold sweep covers one high cap on the routed-menu lane of eight configurations.The sweep leaves the episode budget and call limit unchanged; provider token-counting differences and default reasoning effort also complicate nominal cap comparisons.Source: Limitations · p. 9
  • Each core capacity cell has one run per configuration, while sentinel repeats show 12.1% disagreement and sweep repeats show 17.5% disagreement at 1,024 tokens.Source: Limitations · p. 9
  • Tasks are synthetic and exactly graded without partial credit, prompts are fixed English texts, and results describe model-provider-route pairs rather than models alone.Because the generator and tasks are public, models trained after release may have seen them.Source: Limitations · p. 9

Ethical Considerations

The paper uses synthetic records and local tools without human participants or external side effects. Its protocol fixes task facts while varying tool menus, workflows, or returned-record encodings, then grades obligations and reports evaluation populations by block.

Ethical Considerations

  • All benchmark records, identities, and states are synthetic; no personal data reaches providers, no human participants or annotators are involved, and tools have no external side effects.Results at fixed harness budgets should not alone decide procurement or deployment.Source: Ethical Considerations · p. 9

A Harness Protocol and Prompts

  • A task family fixes the request, facts, answer, evidence rule, and tools while a deterministic compiler varies one pressure at a time.Figure 5 illustrates V menu growth, W workflow growth, and D reencoding of identical facts.Source: A Harness Protocol and Prompts · p. 13, Figure 5 · p. 14
  • The interface-guide prompt explains raw, typed, and linked record fields and provenance, while the workflow prompt adds checks before each request and before answering.No other prompt text differs between those conditions.Source: A Harness Protocol and Prompts · p. 13

B Obligations, Grading, and Audits

  • The grader checks nine obligations in order, and a termination-aware analysis label distinguishes completion-cap truncation before any call from selection failure.The assigned label uses call attempts and error codes without changing the grader’s labels.Source: B Obligations, Grading, and Audits · p. 14
  • Table 3 pairs each episode’s first unmet obligation with its terminal flag; all 849 “selection” labels belong to episodes without a call.Of 20,408 exactly graded episodes, 3,725 are failures.Source: Table 3 · p. 15

C Evaluation Panel

  • Evaluation blocks use different populations and success measures, including exact VTS for capacity and repair blocks, ITT success for Discovery, and ToolSandbox’s evaluator for external blocks.The 21 configurations are managed inference endpoints, with one served through an FP4-quantized route.Source: C Evaluation Panel · p. 15

D Statistical Analysis and Capacity Sensitivity

Across configurations, visible-menu and compact-encoding contrasts are statistically distinguishable under the reported resampling analyses, while workflow pressure is not. The analysis also reports organization-weighted effects and compares routed and fixed menu lanes.

D Statistical Analysis and Capacity Sensitivity

  • Main effects average four paired differences and interactions average two differences-in-differences; compact and typed use checkpoint contrasts against the low, raw reference.Curvature at midpoints and the three-way interaction are exploratory, not among the eight tests.Source: D Statistical Analysis and Capacity… · p. 15
  • Organization-equal reweighting gives −15.3 pp for the visible menu, 3.8 pp for workflow pressure, and −4.7 pp for compact encoding.Source: D Statistical Analysis and Capacity… · p. 16
  • The configuration-resampled visible-menu contrast is [−19.8, −5.7] (Holm p = .011), workflow pressure is [−4.6, 6.5] (Holm p = 1.000), and compact encoding is [−5.0, −2.5] (Holm p <.001).For compact encoding, the exact sign-flip test over 12 intent clusters gives p = .0098 before correction and Holm p = .068.Source: D Statistical Analysis and Capacity… · p. 15, Table 6 · p. 17
  • Table 6 reports paired within-family VTS differences over 21 configurations, with 95% cluster-bootstrap intervals across families, intents, and configurations and an exact sign-flip test over intents.Intervals are unadjusted percentile intervals, while p values use the centered bootstrap and may disagree near zero.Source: Table 6 · p. 17
  • Over 3,024 paired episodes, the routed-menu lane exceeds the fixed-menu lane by 2.4 pp [1.1, 3.8], while using 4,699 fewer tokens and 47.7 fewer schema exposures per episode.Source: D Statistical Analysis and Capacity… · p. 17

E Completion-Cap Crossover and Sweep

The completion-cap sweep shows that removing truncation removes most of the menu penalty in truncation-heavy configurations, while the crossover at four times the cap leaves truncation and the penalty largely intact. The section also reports results for call-budget and repair tests, tool discovery, and whether failure records identify assigned pressures.

E Completion-Cap Crossover and Sweep

F Frontier and Structural Holdout

G Discovery Details

  • Under flat retrieval, ITT success falls 79.6 pp from R = 32 to R = 2,048, and the connector-first router recovers 58.1 pp of that loss.The router’s advantage over configuration switches is by construction; when both retrievers return the required tools, it exceeds flat retrieval for 1 of 21 configurations.Source: G Discovery Details · p. 20

E.1 Sixteenfold Completion-Cap Sweep

  • −43.7 pp at 1,024 tokens versus −2.7 pp at 16,384: the visible-menu contrast shrinks by 41.0 pp in six truncation-heavy configurations as truncation falls to at most 1.4%.The cap interaction is 41.0 pp [36.2, 45.5], and a small penalty remains at the higher cap.Source: §E.1 · pp. 19–20, Table 9 · p. 21

H Failure Localization Audit

  • 54.8% balanced accuracy from call counts alone exceeds every tested failure-record variant, and no paired gain interval for richer records excludes zero.The joint record reaches 48.9%; its paired gain over the first unmet obligation is −4.4 pp [−15.0, 9.7].Source: H Failure Localization Audit · pp. 21–22

F Frontier and Structural Holdout

H Failure Localization Audit

  • For 83 failed rows from packets with several failures, the decoder reaches 62.4% balanced accuracy, but correlated rows make these metrics descriptive.It recovers 3 of 26 returned-state cases.Source: H Failure Localization Audit · p. 22

I Repair and Decision Details

Only the router and controller beat every configuration switch on success in the repair comparison; public-information policy results favor switching in the pilot. In ToolSandbox, larger menus add tokens more clearly than they reduce success, and the paper compares its design with related benchmarks.

I Repair and Decision Details

  • Only the router and controller beat every configuration switch on success: 21/21 incumbents for the router and 17/17 for the controller.The router’s result follows by construction because flat retrieval does not depend on the model; the controller issues the calls itself.Source: Table 12 · p. 23, I Repair and Decision Details · p. 24
  • The routed menu has assigned loss L = 17.5 pp [14.9, 20.2] and selectivity Σ = 4.9 pp [2.2, 7.5], while workflow and interface-guide gains are near zero.The workflow prompt has L = 2.4 pp [−0.4, 5.0] and Σ = 0.0 pp [−2.4, 2.4]; the interface guide has L = 1.5 pp [−0.1, 3.1] and Σ = 0.7 pp [−1.7, 3.1].Source: I Repair and Decision Details · p. 22, Table 12 · p. 23

J Public-Information Policy Pilot

  • In the public-information pilot, the fixed switch succeeds on 99.0% of blocks versus 97.9% for trace selection; after failed probes, the switch reaches 99.5%.The pilot covers 192 blocks; trace selection costs USD 4.49 per 1,000 conditions versus USD 4.62 for probing then switching.Source: J Public-Information Policy Pilot · p. 24, Table 13 · p. 25

K External Environment

  • Across five configurations, ToolSandbox success and VTS are negatively rank-correlated (ρ = −0.8), with both reported intervals below zero.The historical ToolSandbox stratum is reported separately and is not pooled with the concurrent stratum.Source: K External Environment · p. 24
  • In ToolSandbox, exposing all tools adds 16,916 tokens [15,021, 18,676], while full success changes by −2.9 pp [−7.0, 0.5].Ten added distractors add 5,322 tokens [4,622, 6,018] and change full success by −0.5 pp [−1.8, 0.8].Source: K External Environment · p. 24, Table 14 · p. 25, Figure 12 · p. 25

L Comparison with Related Benchmarks

M Reproducibility, Verification, and Cost

The release includes code, data, frozen plans, analysis code, and manifests, with verification checks reported for saved results. The paper also reports an aggregate cost for the scored conditions.

M Reproducibility, Verification, and Cost

Related papers

Loading 2610.21098v1…