PaperlayerView exact revision on arXiv

Source-linked AI summary

SHarP: Saliency-based Pruning of Agent Harnesses

Xinyi Gao, Qiucheng Wu, Kaizhi Qian, Handong Zhao, Shiyu Chang, Yang Zhang

arXiv:2610.04178v1cs.AI

TL;DR

Iterative harness refinement can increase complexity and cost, while the redundancy of added modules remains unclear. SHARP uses single-module ablations to guide pruning by performance and efficiency saliency. Across four harnesses, pruning maintained comparable performance with at least 10% lower token cost; the scope for pruning varied across settings.

  • Problem

    Harnesses grow through iterative refinement, but it is unclear which modules are redundant, and pruning must account for both performance and token cost.

    Source: §1 · pp. 1–2

  • Method

    SHARP estimates module saliency with single-module ablations and uses the scores to guide pruning while protecting modules important to the other objective.

    Source: §1 · p. 2

  • Results

    Across four harnesses, pruning maintained comparable performance while reducing token cost by at least 10%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pruning can reduce redundant actions and input context, but can also remove functionality that helps agents complete tasks efficiently.

    Source: §5 · p. 10

    Single-module ablations hold model configuration, resource budgets, and evaluation conditions fixed within each setting.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Agent harnesses are systems that coordinate model calls, tool use, and task execution to help large language models complete complex tasks. To meet task requirements and address failures, these systems are often iteratively refined by amending and patching their instructions, tools, and workflows, continuously increasing harness complexity. It is therefore unclear whether some resulting harness modules are redundant, introducing substantial token overhead with little, if any, performance gain. Inspired by neural network pruning, in this paper, we study harness pruning as a means of striking a better balance between task performance and token cost. We propose SHarP (Saliency-based Harness Pruning), a simple yet effective pruning strategy based on the saliency of each harness module with respect to performance and efficiency. Specifically, we first identify tools, instructions, and supporting mechanisms as components that can be individually disabled. We then estimate the saliency of each module by ablating it and assessing its task performance and token cost relative to the full set of single-module ablations. Modules with the smallest contribution to performance or largest computational overhead are subsequently pruned. Our evaluation across various harnesses on held-out validation sets reveals a surprising finding: most harnesses that we studied are highly redundant and can maintain comparable performance and efficiency even after a substantial portion of their modules are pruned. Our pruning approach and empirical findings provide new perspectives on agent harness design and optimization.

1 INTRODUCTION

Harnesses grow through iterative patches, but the resulting components may add token costs without improving performance, and no systematic method quantifies or removes that redundancy. SHARP estimates module contributions through single-module ablations and uses saliency to guide pruning; across four harnesses, pruning maintained comparable performance with at least 10% lower token cost.

1 INTRODUCTION

  • Iterative harness refinement can add instructions, tools, examples, and processing steps, increasing harness complexity and computational and token costs.Prior work noted that adding harness mechanisms does not necessarily improve task performance.Source: §1 · p. 1
  • Existing work had not provided a systematic way to quantify harness redundancy or remove redundant components.Earlier studies recognized that added mechanisms may not improve performance or explored restricting access to context and tools.Source: §1 · pp. 1–2
  • Harness pruning must consider both task performance and token cost, while avoiding exhaustive evaluations that consume substantial time and inference resources.Removing a task-completion mechanism, for example, may lead to extra model calls after a task is complete.Source: §1 · p. 2
  • SHARP estimates each module’s performance and efficiency saliency with a single-module ablation, then uses those scores to guide efficiency- or performance-oriented pruning.Efficiency-oriented pruning protects modules with high performance impact; performance-oriented pruning protects modules with high efficiency impact.Source: §1 · p. 2
  • Across four harnesses, pruning maintained comparable performance while reducing token cost by at least 10%, and two harnesses improved task performance while reducing cost.These validation-set results indicate substantial redundancy in the evaluated harnesses.Source: §1 · p. 2

2 RELATED WORK

Prior work studies harness construction and refinement, efficiency controls, and pruning in neural networks and agent workflows. The remaining question is how to prune heterogeneous modules in an existing harness based on both task performance and token cost.

2 RELATED WORK

  • Harness construction searches executable programs for effective reasoning and tool-use combinations, while refinement uses execution traces and feedback to revise components or runtime interfaces.Other approaches generate a harness for each task or adapt interfaces to recurring failures.Source: §2 · p. 3
  • Harness-efficiency work restricts tools or context, compresses interaction histories, or searches configurations while accounting for performance and cost.Some configuration-search work also accounts for noisy evaluations and the costs of both search and deployment.Source: §2 · p. 3
  • Neural-network and agent-workflow pruning remove parameters, larger model components, messages, links, or agents, while prior harness ablations show added execution structure does not consistently improve outcomes.How to prune heterogeneous existing-harness components according to both performance and token cost remained open.Source: §2 · p. 3

3 METHODS

SHARP identifies removable harness modules, estimates their performance and efficiency saliency with single-module ablations, and evaluates saliency-guided pruned configurations on held-out tasks. The method defines harness configurations, two pruning priorities, and a matched evaluation procedure.

3 METHODS

  • SHARP proceeds through module identification, saliency computation on a development set, saliency-guided pruning, and evaluation on a validation set.It considers both efficiency-oriented and performance-oriented pruning strategies.Source: §3 · p. 3

3.1 MODULE IDENTIFICATION AND HARNESS CONFIGURATIONS

  • Modules follow implementation boundaries, with tightly coupled elements grouped together; pruning disables each module according to its function, and A(S) retains modules in S while A(∅) is the all-pruned configuration.For example, pruning a search tool removes its instructions and access, while pruning history summarization means keeping the full history.Source: §3.1 · p. 4

3.2 MODULE SALIENCY COMPUTATION

  • On development tasks, each single-module ablation records a task-dependent success score in [0, 1] and token cost across model calls.Model configuration, resource budgets, and evaluation conditions are held fixed across ablations within each setting.Source: §3.2 · p. 4

3.3 SALIENCY-GUIDED PRUNING

  • Efficiency-oriented pruning protects modules considered crucial for performance and orders the remaining modules by efficiency saliency for successive pruning.The method also considers performance-oriented pruning, which prioritizes task performance.Source: §3.3 · p. 5

3.4 PRUNING ANALYSIS

  • Pruned configurations are evaluated on validation tasks disjoint from development, under matched conditions, and compared with the full and all-pruned harnesses on performance and token cost.By default, each configuration receives three trials per task.Source: §3.4 · p. 5

4 EXPERIMENTS

Across five evaluated settings, many pruned configurations retained similar observed performance, but the extent and cost of redundancy varied by harness and domain. The experiments compare pruning strategies and examine mechanisms behind both efficiency gains and cost increases.

4.1 EXPERIMENTAL SETUP

  • The evaluation spans five settings across four harnesses, uses development tasks for saliency estimates, and measures validation-set performance and token cost.Main results use three trials per task by default; reported performance is success rate for GAIA, LitQA2, and τ 2-bench, and mean per-task match rate for DeepPlanning-Shopping.Source: §4.1 · pp. 5, 7

4.2 MAIN RESULTS

  • Across five settings, many pruned configurations retain similar observed performance, but deeper pruning can lower performance or raise token cost.OpenHands reached its highest GAIA success rate, 35.00%, with 24 of 27 modules pruned; with all 27 pruned, success fell to 14.44%.Source: §4.2 · p. 7

4.3 COMPARISON OF PRUNING STRATEGIES

  • On DeepPlanning-Shopping, performance-oriented pruning reaches 82.5% match at S′19 with 5.9% lower mean token cost, while efficiency-oriented S20 reaches 76.2% match with a 39.0% reduction.On retail, efficiency-oriented pruning reaches the highest observed success rate at S43 (87.5%); performance-oriented pruning’s highest is the all-pruned configuration (85.0%). Relative benefits depend on harness and domain.Source: §4.3 · pp. 7–8

4.4 QUALITATIVE ANALYSIS

  • Removing JIT agentfold’s shopping-cost guidance preserved the 100% match rate while reducing calls from 43 to 14 and total token cost by 74%.The full harness made 26 further searches and 26 product-detail queries after assembling the correct cart, without changing the selected products.Source: §4.4 · p. 9
  • In PaperQA2, removing completion or evidence-retrieval modules can increase cost by prolonging execution or making the agent process all collected passages.Without completion, the agent continues searching and generating answers until the step budget is exhausted; without retrieval, it summarizes passages that would have been filtered out.Source: §4.4 · p. 9, Figure 4 · p. 9

5 CONCLUSION

SHARP uses saliency-guided pruning to analyze harness redundancy; smaller configurations can retain similar task performance while reducing token cost. Pruning scope varies, and removing more modules does not always improve efficiency.

5 CONCLUSION

  • Single-module ablations guide SHARP pruning, and smaller harness configurations can retain similar task performance while reducing token cost across diverse settings.The scope for pruning varies across harnesses and task domains.Source: §5 · p. 9
  • Pruning can reduce redundant actions and input context, but it can also remove functionality that helps agents complete tasks efficiently.The conclusion suggests examining modules' contributions to performance and efficiency alongside efforts to add capabilities.Source: §5 · p. 10

A.1 HARNESS MODULES

The appendix catalogs configurable modules across the evaluated harnesses and clarifies what remains available when candidate modules are disabled.

A.1 HARNESS MODULES

  • All-pruned configurations retain different capabilities: LIFE-harness keeps native domain tools and policy, JIT keeps shopping APIs and its basic action protocol, and PaperQA2 answers by direct model call without tools.The benchmark shopping APIs and final-answer interface remain available in every JIT configuration.Source: §A.1 · p. 13
  • Candidate inventories include 18 independently configurable PaperQA2 modules, 23 JIT agentfold modules, and 27 OpenHands modules spanning tools, instructions, and context management.The JIT inventory is for DeepPlanning-Shopping, and its shopping guidance varies with task difficulty.Source: §A.1 · pp. 13–14, Table 4 · p. 14
  • The inventories include task-specific guidance such as shopping requirements, price strategy, cart verification, and output instructions.For LIFE-harness, examples include reservation, payment, policy, and workflow guidance among its separately configurable modules.Source: §A.1 · pp. 14–16
  • LIFE-harness has 44 airline and 48 retail modules grouped into checks, tool-description hints, result annotations and alerts, retrieved skills, and agent instructions.Each hint or annotation is separate from its underlying tool; the native domain tools and policy remain available when these modules are disabled.Source: §A.1 · p. 15

A.2 IMPLEMENTATION DETAILS

The implementation details specify the model and harness setups, the token accounting procedure, and the task-level shopping match metric used in the experiments.

A.2 IMPLEMENTATION DETAILS

  • For τ 2-bench retail, two of 50 sampled development tasks are excluded because their evaluation outputs could not be parsed, leaving 48 evaluable tasks.Table 7 summarizes task counts and candidate-module counts.Source: §A.2 · p. 18, Table 7 · p. 20
  • Experiments serve the FP8 Qwen3.5-122B-A10B checkpoint through vLLM with thinking disabled; most runs use H100 GPUs, while some use A100s.OpenHands uses temperature 0.7 and top-p 0.8, while the other harnesses use temperature 0.Source: §A.2 · pp. 18, 20
  • The harness implementations use benchmark-specific pipelines, including OpenHands' GAIA pipeline, JIT's fixed agentfold seed harness, PaperQA2's fixed local corpus, and LIFE-harness' native τ 2-bench environments.JIT is evaluated directly without the repository's harness optimization process; LIFE-harness uses benchmark assertion checks.Source: §A.2 · p. 20
  • Reported token cost sums input and output tokens across recorded model calls for each task, including unsuccessful attempts, and measures token usage rather than elapsed execution time.Table 9 specifies which calls are included for each harness.Source: §A.2 · p. 20, Table 9 · p. 21
  • The shopping match rate counts matched distinct product IDs and coupon entries against the corresponding reference totals, with product matches requiring name and quantity agreement.Extra product IDs in the cart are recorded separately and do not reduce the score; per-task match rates are averaged across tasks and trials.Source: §A.2 · pp. 20–21

A.3 STATISTICAL TESTS FOR MODULE SALIENCY

Module saliency is assessed with paired one-sided tests on development-task outcomes, comparing each single-module ablation against a task-matched median reference.

A.3 STATISTICAL TESTS FOR MODULE SALIENCY

  • Performance and efficiency saliency are computed using two one-sided paired t-tests on development tasks, pairing each single-module ablation with the same task's median reference.The task is the unit of analysis, and each test is implemented as a one-sample t-test on paired differences.Source: §A.3 · p. 21
  • The performance test assesses whether disabling a module lowers task scores, while the efficiency test assesses whether it raises token cost.Source: §A.3 · p. 21
  • The tests estimate mean and variance across development-task paired differences; when those differences are constant, a sign-based convention assigns p-values of 0 or 1.If all differences are zero, the assigned p-value is 1.Source: §A.3 · p. 21

A.4 MODULE SALIENCY RESULTS

This section reports single-module ablation results and the p-values used to estimate performance and efficiency saliency across agent harnesses.

A.4 MODULE SALIENCY RESULTS

  • The ablation results report task performance and token cost after disabling individual modules, alongside p-values for performance and efficiency saliency.The tests are one-sided paired t-tests, and the tables use per-task median references.Source: §A.4 · p. 21, Table 10 · p. 22, Table 11 · p. 22
  • JIT agentfold and PaperQA2 ablations each cover 50 development tasks, with match rate or success rate and mean token cost reported.Each table uses one trial per task.Source: Table 10 · p. 22, Table 11 · p. 22
  • OpenHands ablations cover 50 GAIA development tasks, while LIFE-harness ablations cover 30 airline and retail development tasks.The tables report success rate and mean token cost, using one trial per task.Source: Table 12 · p. 23, Table 13 · p. 23, Table 14 · p. 24

A.5 ADDITIONAL PRUNING RESULTS

The appendix reports additional pruning configurations and validation results across JIT agentfold, PaperQA2, OpenHands, and LIFE-harness, including full-range plots for two harnesses.

A.5 ADDITIONAL PRUNING RESULTS

  • JIT agentfold efficiency-oriented pruning entries include P=76.88% with T=219.1 and P=68.49% with T=245.9.P is mean match rate and T is mean token cost (10^3); the table covers 30 validation tasks with three trials per task.Source: Table 15 · p. 25, §A.5 · p. 26
  • The protected JIT agentfold configuration reports 63.28% mean match rate and 145.7 mean token cost (10^3).The configuration retains the protected module set; the table covers 30 DeepPlanning-Shopping validation tasks with three trials per task.Source: Table 15 · p. 25, §A.5 · p. 26
  • Additional efficiency-oriented validation tables cover PaperQA2 on LitQA2, OpenHands on GAIA, and LIFE-harness on airline and retail tasks.The retail results also include a performance-oriented pruning table; the airline protected configuration is all-pruned because its protected set is empty.Source: Table 17 · p. 26, Table 18 · p. 26, Table 19 · p. 27
  • Figure 5 shows full-range PaperQA2 and JIT agentfold plots, with stars marking full harnesses, colors indicating pruned modules, and lines tracing the Pareto frontier.It includes PaperQA2 S3 and JIT agentfold S0, which are omitted from the zoomed-in Figure 3 panels.Source: §A.5 · p. 28

A.6 ADDITIONAL QUALITATIVE ANALYSIS

The qualitative analysis examines how pruning affects workflow progression, evidence processing, and tool use, including module dependencies and alternative routes.

A.6 ADDITIONAL QUALITATIVE ANALYSIS

  • Without PaperQA2’s evidence-gathering tool, an inspected trajectory repeatedly searches without accumulating evidence contexts, then invokes fallback evidence gathering after exhausting its execution budget.The fallback produces an answer; the observation illustrates that document access alone may not be enough for workflow progress.Source: §A.6 · pp. 28–29
  • PaperQA2’s automatic evidence-gathering workload depends on whether retrieval remains enabled, because retrieval bounds the chunks processed while disabling it exposes all available chunks.This is an example of retained modules jointly determining execution work.Source: §A.6 · p. 29
  • In OpenHands, removing a web tool accompanies increased use of alternatives: browser and fetch calls rise without search, search calls rise without the browser, and terminal use rises when all three web tools are disabled.These observed patterns suggest that tool availability can shape the agent’s response to pruning.Source: §A.6 · p. 29

Related papers

Loading 2610.04178v1…