PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses

Cen, Zhao, Haibo Ruan, Wenjie Chen, Pei-fen Tu, Usman Abbasi, Joel Hesch

arXiv:2609.05736v1cs.AI

TL;DR

The paper asks whether fixed-model multi-turn tool agents can be improved through resource-bounded harness changes without retraining. It proposes reliability-centered harness selection with prompt and tool-boundary middleware optimization, finding that PRISM achieves the strongest selected-harness profile across three benchmarks while motivating reliability reporting alongside average lift.

  • Problem

    The paper studies whether runtime harness optimization can improve fixed-model tool-agent behavior while preserving deployment constraints such as latency and runtime cost.

  • Method

    The paper evaluates budgeted harness selection using RelLift95 and compares prompt-only with failure-routed prompt and tool-boundary middleware optimization through PRISM.

  • Results

    PRISM achieves the strongest selected-harness profile across BFCL multi-round, τ 2-Retail, and τ 2-Telecom, with the highest mean held-out lift and positive RelLift95 on all three benchmarks.

  • Takeaways & Limitations

    Reliability of the selected harness should be reported alongside average held-out lift because some optimizers can select brittle updates despite occasionally finding large gains.

  • Takeaways & Limitations

    The evaluation covers three controlled offline benchmarks and does not capture production traffic, changing users, long-running memory, adversarial users, or business-specific safety policies.

Abstract

from arXiv · show

LLM tool agents can be improved without retraining by modifying the runtime harness around a fixed model: prompts, tool interfaces, middleware, state handling, and recovery logic. We study this setting as resource-bounded harness selection for fixed-model multi-turn tool agents, with the search surface scoped to prompts and tool-boundary middleware: edits are guarded intercepts at the tool boundary, not arbitrary rewriting of agent execution logic. Our optimizer-agnostic protocol reports mean held-out lift, worst-condition lift, repeatability, logged cost diagnostics, and RelLift95(B), a conservative estimate of the held-out gain of the harness selected under budget B. We instantiate the protocol with prompt-only and prompt-plus-middleware optimizers, including PRISM, which clusters failures and routes repairs to prompt, tool-boundary middleware, or joint edit surfaces within a Pareto search. On BFCL multi-round, tau2-Retail, and tau2-Telecom, PRISM obtains mean held-out lifts of 14.2, 14.9, and 10.1 percentage points and positive empirical RelLift95 on all three benchmarks, and a component ablation attributes the margin chiefly to failure-surface routing and the edit-pattern constraint. Across optimizers, the results show that some search procedures can occasionally find large gains but still choose brittle updates, so the reliability of the chosen harness should be reported alongside average held-out lift.

1 Introduction

The paper frames harness optimization as improving fixed-model, multi-turn tool agents under deployment constraints without changing model weights. It narrows the searchable surface to prompts and tool-boundary middleware and asks whether automatic search reliably improves held-out behavior.

  • Problem and scope: Multi-turn tool agents must track dialogue state, follow policies, call APIs correctly, recover from missing information, and leave environments in intended states.τ2-bench and BFCL capture complementary aspects of dynamic interaction, stateful execution, and executable function calling.
  • Problem and scope: A harness is the runtime control layer around a fixed model, including prompts, tool interfaces, context, state, middleware, recovery, and evaluation hooks.The optimized surface is scoped to prompts plus tool-boundary middleware, where failures are locally observable and checkable.
  • Research question: The central question is whether automatic harness search can improve a fixed model reliably on held-out data rather than merely raise one benchmark score.The paper studies this question within a precisely bounded prompt-and-middleware search surface.
  • Approach: PRISM factorizes prompt and middleware updates, routes repairs to locally checkable surfaces using clustered failure causes, and selects candidates by gate pass rate and reliability.Its search builds on a GEPA-style evolutionary and Pareto-search skeleton.

2 Background and Related Work

Related work establishes harnesses and prompts as optimization surfaces around fixed models, while other methods edit agent programs or harness source. The paper positions its contribution as tool-boundary repair with a deployment envelope and reliability evidence.

  • Harness optimization: Prior work treats harness failures as training data for hill climbing and exposes profile-specific harness surfaces with holdout evaluation or task gains.BetterHarness uses regression checks before deployment, while Deep Agents profiles report gains on τ2-bench-style tasks.
  • Prompt optimization: Prompt-oriented methods include instruction optimization, evolutionary prompt populations, textual feedback propagation, language-model program compilation, and trajectory-reflective Pareto search.The paper uses GEPA and MIPROv2 as baselines rather than every earlier ancestor.
  • Scope distinction: A separate line edits agent code or harness source, whereas this paper studies tool-boundary repair around a fixed model and tool API.The related works report differing evidence, including validation lift, rollout efficiency, regression checks, and single final scores.
  • Reliability motivation: Repeated search-and-release motivates reporting more than a single validation lift because successive batches of failure reports can differ.The paper places this concern within a deployment envelope inside the score itself.

3 Evaluation Protocol and Metrics

The protocol evaluates black-box harness optimizers on disjoint repair, gate, and held-out scorecard data. It reports effectiveness, stability, repeatability, cost, and a budgeted lower-tail estimate of selected-harness lift.

  • Protocol: Each optimizer is treated as a black box that returns a candidate harness after declaring pre-scorecard search cost, followed by held-out scorecard evaluation.The final scorecard evaluation is excluded from the search-cost accounting.
  • Data splits: The repair split proposes and tests edits, the gate split accepts candidates and selects the final harness, and the scorecard split is held out until final reporting.Baseline and returned harnesses are evaluated on the same scorecard cases, making lift a paired comparison.
  • Metrics: The benchmark-native score S_b,m(h, D) averages per-instance scores when available, while crashes, timeouts, invalid actions, and protocol violations follow benchmark failure rules.This defines the interface for comparing harnesses across benchmark-specific metrics.
  • Stability and repeatability: WorstLift_m is the lowest mean lift across pre-specified conditions, while RepRate_δ,m is the fraction of runs achieving lift at least δ.RepRate_0,m, or RR0, measures the fraction of runs with positive held-out lift.
  • Budgeted reliable lift: RelLift is an empirical lower-tail estimate of selected held-out lift under budget B, obtained by bootstrapping optimizer runs and selecting by the pre-scorecard statistic G_i.It is a deployer-side diagnostic, not a coverage guarantee over future scorecards, simulator seeds, or repair distributions.

4 Experiments

The experiments compare prompt-only and prompt-plus-middleware harness optimizers under shared rollout budgets, disjoint data roles, and held-out scorecard evaluation. PRISM combines failure routing, constrained tool-boundary edits, evolutionary search, and Pareto selection over gate pass rate and reliability.

  • Compared optimizers: Four optimizers are compared through prompt-only and prompt-plus-middleware variants, with unmodified published forms and symmetric surface-toggled counterparts.-PO variants edit prompts only, while -MW variants may edit middleware at the tool boundary.
  • PRISM: PRISM routes clustered failures to prompt, middleware, or joint mutation slots, while restricting middleware changes to three tool-boundary edit patterns.Its evolutionary loop evaluates children on repair and gate splits and retains a Pareto frontier over gate pass rate and reliability.
  • Budget and diagnostics: Search budgets count inner-model task rollouts before scorecard evaluation, while monetary cost is logged separately and each rollout must satisfy benchmark-specific resource envelopes.Rollouts exceeding wall-clock, turn, or token caps, or entering repeated tool-call loops, receive zero score.
  • Reliable lift: RelLift95(B) is estimated from at most 16 observed runs using 5,000 bootstrap resamples, and cross-optimizer comparisons rely on point estimates unless intervals are available.The estimator’s 5th-percentile interpretation is read alongside WorstLift and RR0, with differences smaller than its own interval not treated as differences.

5 Results

Results show that reliable harness selection depends on both the available edit surface and the optimizer’s ability to choose robust updates under pre-scorecard information. PRISM performs strongly across benchmarks, while unconstrained middleware and peak-oriented search can increase selection risk.

  • Main results: PRISM-MW ends up strongest on every task, with its margin attributed chiefly to failure-surface routing and the edit-pattern constraint.The component ablation attributes the margin to routing and constrained middleware edits rather than in-loop gate access.
  • Prompt versus prompt-plus-middleware: Middleware improves every optimizer receiving the pattern-guided surface, but unconstrained BH middleware falls below prompt-only BH on both τ 2 tasks.The three supported middleware patterns are silent correction, error blocking for retry, and prerequisite blocking before environment mutation.
  • Measurement caveat: RelLift95 is an empirical lower-tail estimate from observed runs, not a guarantee, so it should be interpreted alongside WorstLift and RR0.Figure 2 and Table 2 use pre-scorecard information for selection; scorecard evaluation remains held out.
  • Lift distribution and selection reliability: 14.2 pp MEANLIFT, 11.0 pp WORSTLIFT, and 12.0 pp RelLift95: PRISM leads on BFCL while all methods reach RR0 = 100%.On BFCL, the comparison is therefore about lift magnitude rather than repeatability.
  • Lift distribution and selection reliability: 10.1 pp RelLift95 on Retail and 7.8 pp on Telecom: PRISM maintains positive selected tails despite a lower peak than BH.BH finds the best individual harnesses on both τ 2 tasks but has WORSTLIFT = −13.5 pp and negative RelLift95.
  • Budgeted reliable lift: PRISM rises early and plateaus highest on τ 2-Retail, whereas Telecom rewards larger budgets for PRISM-PO and MIPROv2.Figure 2 evaluates pre-scorecard selection reliability rather than oracle best-of-N performance, using the inner-rollout cost convention.

6 Ablation Study

The ablations isolate PRISM’s routing, gate, cross-generation failure tracking, and middleware constraints, while model-role sensitivity shows that both execution and proposal models affect reliable harness optimization. Stronger or better-matched models improve different parts of the search, and lower-cost models can remain useful for exploration.

  • 6.1 Component ablation: Each ablation switches off exactly one PRISM component while holding the remaining protocol and budget fixed.Variants include no routing, final-only gate access, no cross-generation failure matrix, and unconstrained middleware editing.
  • 6.1 Component ablation: GEPA-MW trails full PRISM despite combining Pareto retention with the middleware surface, indicating that the full configuration contributes beyond frontier retention.The comparison is made against PRISM-MW under the component-ablation framing.
  • 6.2 Model-role sensitivity: GPT-5.4 reaches BFCL MaxSC 73.6, but GPT-5.4-mini matches its mean lift with a stronger lower tail at roughly one-fifth the logged cost.On τ 2-Retail, GPT-5.4-mini leads every lift and reliability statistic, while GPT-5.4 attains the highest absolute gate score, MaxSC 59.0.
  • 6.2 Model-role sensitivity: Opus 4.7 is strongest across BFCL outer-model lift and reliability measures, while τ 2-Retail separates mean quality from lower-tail reliability across proposers.Sonnet 4.6 has the largest Retail mean and maximum lift, whereas GPT-5.5 has the strongest lower-tail reliability.
  • 6.2 Model-role sensitivity: Lower-cost proposers support exploration, but stronger proposers are safer when the selected harness must be reliably better rather than occasionally better.The practical trade-off depends on whether the objective emphasizes discovery or dependable selected-harness performance.

7 Conclusion

The paper presents reliability-centered, resource-bounded harness optimization for fixed-model tool agents and reports stronger selected-harness reliability for PRISM than compared baselines. Its evidence remains bounded by controlled offline benchmarks, stochastic small scorecards, and constrained middleware edits.

  • Conclusion: PRISM combines budgeted reliable-lift evaluation with failure-routed prompt and tool-boundary middleware search.The reported margin is attributed chiefly to failure-surface routing and the edit-pattern constraint.
  • Scope: The evaluation covers BFCL multi-round, τ 2-Retail, and τ 2-Telecom, spanning function calling, service workflows, policy following, and tool-mediated state changes.These remain controlled offline settings without production traffic, changing user populations, long-running memory, adversarial users, or business-specific safety policies.
  • Limitations: The baselines use strong human harnesses, while the fully controlled test seeding search with the best prompt-only-optimized harness remains open.This limits conclusions about optimization from an already prompt-optimized starting point.
  • Limitations: The τ 2 scorecards are small held-out subsets with stochastic user simulation, so table intervals and RelLift95 values are empirical estimates rather than guarantees.Broader validation would require more domains, larger scorecards, more split seeds, and live deployments.
  • Limitations: Cross-arm τ 2-Telecom comparisons carry an orchestration caveat because PRISM-PO and PRISM-MW used four genuine simulator trials outside the benchmark-native pass4 path.The other Telecom arms used the benchmark-native scoring path.
  • Limitations: Middleware optimization increases both power and risk: unconstrained edits can change tool semantics, while the three-pattern vocabulary excludes cross-call coordination, retries, asynchronous state, and control-flow changes.The evidence supports middleware optimization where failures are locally checkable at the tool boundary.

A Model-Role Ablation Tables

The sensitivity grids independently rerun PRISM across inner and outer models, distinguishing optimization outcomes from the main comparison and treating costs as logged search diagnostics.

  • Model-role sensitivity: Tables 4 and 5 are independent reruns, so they provide additional sensitivity evidence rather than replacements for the main comparison.Their costs follow Appendix F and report Cost/run (all), including outer-proposer calls.

B Evaluation Protocol Details

The protocol separates repair, gating, and held-out scorecard roles, evaluates optimizer runs under fixed resource budgets, and selects candidates using gate performance and reliability rather than score alone.

  • Evaluation protocol: The scorecard split is hidden during search and excluded from analysis, mutation, crossover, frontier updates, and pre-scorecard selection.This preserves held-out evaluation for final reporting.
  • Failure routing: PRISM clusters failures by root cause and routes each cluster to prompt-only, middleware-only, or joint repair surfaces.The analyst receives case identifiers, stuck types, tool-call sequences, and truncated state differences or error messages.
  • Mutation design: PRISM runs three surface-constrained mutation slots in parallel, while PRISM-PO forces all slots to prompt-only edits.This design separates search strategy from search surface.
  • Evaluation protocol: Each benchmark–optimizer pair uses repeated split seeds and a reliability-screened pool of at most 16 runs.The design targets four split seeds times four independent repeats per seed.
  • Reliable lift: RelLift95(B) is estimated by bootstrapping budgeted selections from the observed reliable-run pool.Scorecard outcomes are not used during search or pre-scorecard selection.
  • Selection: The optimizer retains candidates on a Pareto frontier using gate score and reliability, with deterministic tie-breaking by run identifier.Repair split score is excluded from dominance so repair–gate trade-offs do not discard candidates that generalize better.

D Prompt and Middleware Examples

The examples show that useful repairs depend on the failure surface: BFCL benefits from structural middleware fixes, τ 2-Retail from lookup and validation logic, and τ 2-Telecom from prompt-level policy guidance.

  • D Prompt and Middleware Examples: PRISM’s three mutation slots route failure clusters to prompt-only, middleware-only, or joint edits, while PRISM-PO disables middleware changes.The slots run in parallel and receive clusters according to routing decisions.
  • D.1 BFCL Example: BFCL requires exact tool-call correctness, so middleware can repair recurring argument-name, type-coercion, and JSON-structure errors before execution.These are structural repairs rather than only instruction-following problems.
  • D.1 BFCL Example: +5.8 percentage points is the BFCL improvement for prompt-only PRISM-PO, versus +14.2 percentage points for full PRISM with middleware.The comparison accompanies examples of schema correction, type coercion, and JSON unwrapping.
  • D.2 τ 2-Retail Example: τ 2-Retail benefits from proactive order lookup and middleware validation of item–order consistency when the model lacks an order ID or confuses multiple orders.Middleware builds an order-to-item map from prior get_order_details responses and returns actionable errors before execution.
  • D.2 τ 2-Retail Example: 8% of τ 2-Retail baseline trajectories across tested models exhibit the item–order confusion pattern described in the middleware example.The backend’s opaque error often leads to repeated retries.
  • D.3 τ 2-Telecom Example: τ 2-Telecom’s observed failures are policy-level: a prompt checklist guides when to continue troubleshooting or escalate, while middleware rewriting is unnecessary.The model already knows the relevant tool calls, so the missing capability is ordered policy guidance.
  • Artifacts: The released artifacts include the PRISM implementation, baseline wrappers, benchmark runners, metric scripts, configurations, and reproducibility commands, subject to benchmark and trajectory-release restrictions.The paper does not release benchmark records, full-trajectory model outputs, or model weights.

E.2 Compute, infrastructure

Experiments use API-based fixed-model evaluation with explicit rollout, concurrency, timeout, optimizer, and run-exclusion settings. Reliability is assessed through repeated runs and bootstrap/subsampling checks of PRISM-MW RelLift95 estimates.

  • Compute and infrastructure: API-based experiments measure computational budget through rollouts, model calls, token usage, and API cost rather than GPU hours.The main configuration uses gpt-5.4-mini for inner execution and claude-opus-4.7 for outer proposals.
  • Execution limits: BFCL and τ2-bench evaluations run up to 8 cases concurrently, with per-rollout timeouts of 5 and 10 minutes, respectively.The outer model is capped at 300 tool-use turns per optimization run.
  • Run handling: Runs with final repair-split reliability below 0.5 are excluded as infrastructure failures, so per-arm denominators can be below 16.The exclusion covers API crashes and rate-limit storms rather than optimizer outcomes.
  • Optimizer settings: PRISM uses 10 generations, 3 mutations per generation, and a population cap of 5, while BetterHarness uses 10 iterations.MIPROv2 uses 10 candidates, with trial and minibatch settings varying between BFCL and τ2-bench domains.
  • Reliability checks: PRISM-MW RelLift95 estimates remain stable under subsampling, with standard deviation reaching at most 2.4 points at n=8 and all confidence intervals excluding zero.The analysis uses percentile-bootstrap 95% confidence intervals and 200 random subsamples at each size.

F Token Tracking, Model Pricing, and Cost Methodology

The paper tracks pre-scorecard model-call costs using conservative rate-card accounting, separating inner-only experimental costs from all search-time model calls. These costs are diagnostics rather than selection criteria and exclude several deployment expenses.

  • Interpretation: Monetary cost is an implementation-level diagnostic rather than the primary fairness constraint or a harness-selection criterion.The final held-out scorecard evaluation is excluded from both reported cost scopes.
  • Cost scopes: Cost/run (inner) sums pre-scorecard inner-model task-rollout calls, while Cost/run (all) also includes outer-proposer calls.Under the main configuration, the reported figures are $103 versus $142 per BFCL run.
  • Token tracking: Logged costs use input and output token counts for each model call made before scorecard evaluation.The set of calls is denoted Mi for optimizer run i, with model-specific token quantities recorded for pricing.
  • Pricing convention: When internal and public standard prices are both available, the calculation uses the larger rate and applies no cached-token, batch, flex, reserved-capacity, or enterprise discounts.For routes without a public rate card, the internal route-card rate is used and marked accordingly.
  • Scope boundary: The cost quantity excludes online serving, infrastructure, human review, and final held-out scorecard evaluation, so it is not a complete end-to-end deployment cost.The rate card is specific to this paper’s implementation-level accounting rather than universal provider pricing.
Loading 2609.05736v1…