PaperlayerView exact revision on arXiv

Source-linked AI summary

Toolcompass: Guiding Tool Trialing, Not Suppressing It

Junlin Fang, Chong Zhang, Do Nguyen-Thanh, Xiaogang Xu, Zhen Fang, Sean Du

arXiv:2609.25678v1cs.AIcs.LG

TL;DR

Tool-use agents must curb wasteful trials without suppressing exploration needed for unfamiliar tools. ToolCompass organizes tool-call representations around shared functions, and experiments report consistent gains across post-training methods, including on OOD tasks.

  • Problem

    Outcome-based post-training leaves wasteful trials unguided, while turn-level supervision may discourage exploration needed when tools are unseen.

    Source: §1 · p. 1

  • Method

    ToolCompass models function classes with vMF distributions and jointly shapes tool-call representations to reduce within-function variation and separate different functions.

    Source: §1 · pp. 1–2

  • Results

    Across GRPO, RFT, and DMPO, ToolCompass consistently improves task success; AppWorld OOD success rises by up to 8.87 percentage points over vanilla GRPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Organizing representations by function transfers experience from seen tools to functionally similar unseen tools and directs exploration away from unrelated alternatives.

    Source: §1 · p. 2, §6 · p. 9

    The setup assumes unseen tools come from held-out domains and may share functions with seen tools; those tools and their annotations are unavailable during post-training.

    Source: §2 · p. 3

Abstract

from arXiv · show

Large language model (LLM) agents must generalize from tools seen during training to unseen tools at deployment. A key challenge is tool trialing, i.e., excessive trials waste the interaction budget, whereas selective trials enable exploration of unfamiliar tools. Existing outcome-based post-training leaves wasteful trials unguided, while turn-level supervision may suppress necessary exploration. We introduce ToolCompass, a post-training framework that guides tool trialing by organizing tool-call representations according to shared functions. Specifically, ToolCompass models each function class as a von Mises--Fisher distribution and jointly reduces intra-function variation across domains and increases inter-function separation. This structure transfers experience from seen tools to functionally similar unseen tools, directing exploration away from unrelated alternatives. ToolCompass requires no ground-truth call traces or unseen-tool access and incurs no inference overhead. Experiments on AppWorld and FTRL show consistent gains across GRPO, RFT, and DMPO. improves AppWorld OOD task success by up to 10.71 percentage points over vanilla post-training and performs best among competitive baselines on both benchmarks.

1 Introduction

Tool-use agents must limit wasteful tool trials without losing exploration needed to use unfamiliar tools. ToolCompass organizes representations by shared function to transfer experience to unseen tools, and improves OOD task success across post-training methods.

1 Introduction

  • Tool trialing must balance reducing unnecessary calls, which consume interaction budget, with exploration that may be essential for unfamiliar tools.On held-out AppWorld applications, MatchTIR makes fewer calls but achieves lower task success than GRPO.Source: §1 · p. 1, Figure 1 · p. 2
  • GRPO-trained representations cluster by domain rather than function, leaving agents to try unrelated tools instead of exploiting similarities between seen and unseen tools.Calls sharing a function remain far apart across domains.Source: §1 · p. 1
  • ToolCompass models function classes with vMF distributions, drawing calls toward function prototypes while separating different prototypes to transfer experience across domains.Its objective is jointly optimized with the original post-training objective.Source: §1 · pp. 1–2
  • ToolCompass consistently improves task success across post-training methods and improves AppWorld OOD success over vanilla GRPO by up to 8.87 percentage points.With Qwen3.5-9B, AppWorld OOD success reaches 70.02%.Source: §1 · p. 2
  • ToolCompass needs function-class assignments for seen tools, but no ground-truth call traces or unseen-tool access, and adds no inference overhead.It can be combined with GRPO, RFT, and DMPO.Source: §1 · p. 2

2 Problem Setup

The setup formalizes post-training on a fixed set of seen tools and deployment on tasks that may require unseen tools. The goal is to transfer function-level knowledge while preserving performance on in-distribution tasks.

2 Problem Setup

  • A tool-use agent interacts with an environment through a trajectory of states, actions, and observations; states include the request, interaction history, and available-tool specifications.Actions may be language responses or tool calls, and observations may include execution results or errors.Source: §2 · p. 2
  • Tools from different domains may share a function, so the setup groups tools into function classes and post-trains on tasks involving a fixed set of seen tools.For example, amazon.search_products and spotify.search_songs both perform Search.Source: §2 · p. 2
  • At deployment, OOD tasks may require tools from held-out domains that share functions with seen tools, while their tools and annotations are unavailable during post-training.The objective is to transfer function-level knowledge to unseen tools while preserving in-distribution performance.Source: §2 · p. 3
  • The representation figure contrasts GRPO's domain-based clusters with ToolCompass's alignment of calls sharing a function across domains.Source: Figure 2 · p. 3

3 Method

ToolCompass shapes tool-call representations by shared function and jointly trains this objective with the original post-training objective. Its design organizes calls into compact, separated function clusters to support transfer between tools.

3.1 Tool-Call Representation

  • Final-token hidden states summarize each tool call and are projected onto the unit hypersphere, where angular similarity represents relationships between calls.The hidden state captures the selected tool, its arguments, and the current interaction state.Source: §3.1 · p. 3

3.2 Modeling Tool Functions with vMF Distributions

  • Each function class is modeled as a vMF distribution around a prototype, with losses aligning same-function calls and separating different-function prototypes.This structure is intended to transfer experience from seen tools to unseen tools with matching functions, guiding trials toward plausible alternatives.Source: §3.2 · pp. 3–4
  • Unlike trace-matching or turn-contribution supervision, ToolCompass guides where the agent trials without prescribing a call sequence or directly penalizing exploration.Outcome-based methods such as GRPO provide trajectory-level feedback but not function-level guidance for individual calls.Source: §3.2 · p. 5

3.3 Joint Optimization with Post-Training

  • The vMF representation-shaping loss is jointly optimized with the original post-training objective, which can use GRPO, RFT, or DMPO.The projection head and prototypes are removed after training, so deployment uses the original policy without an auxiliary scoring module or additional inference pass.Source: §3.3 · p. 5

4 Experiments

Across AppWorld and FTRL, ToolCompass is evaluated with multiple post-training objectives, model scales, and competitive baselines; analysis examines its design and trialing behavior. It improves results across objectives and benchmarks, while retaining more useful trials and reducing unproductive ones relative to GRPO.

4.1 Setup

  • Experiments use AppWorld and FTRL with Qwen3.5-4B and Qwen3.5-9B; unseen tools and annotations are unavailable during post-training, and performance is measured on ID, OOD, and full test sets.The study compares prompting agents, post-training objectives, turn-level methods, tool-use RL, representation learning, and OOD-generalization baselines.Source: §4.1 · p. 5

4.2 Main Results

  • ToolCompass improves every reported ID, OOD, and total score across both benchmarks and model scales; on Qwen3.5-4B, GRPO total-score gains are 9.18 points on AppWorld and 10.79 on FTRL.With RFT, gains are 3.42 and 5.66 points, and with DMPO they are 3.65 and 5.85 points, respectively.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Analysis

  • Removing any loss component degrades performance; removing LvMF drops success from 66.44% to 57.21%, while variation and separation losses also contribute.Performance peaks at τ=0.4 and d=128, remains competitive across broad ranges, peaks at layer 8, and stays above vanilla GRPO at every tested layer.Source: §4.3 · pp. 6–7, Figure 5 · p. 6

4.2 Main Results

  • ToolCompass achieves the best OOD and total scores across both benchmarks and model scales; on AppWorld with Qwen3.5-9B, its 72.36% total task success rate exceeds LOOP's 68.26% by 4.10 points.A few proprietary models remain competitive on AppWorld ID, but ToolCompass surpasses all prompting agents on every OOD and total score.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Analysis

  • EMA prototypes perform best among the tested estimators, and cosine similarity reaches 66.44% task success versus 60.28% for negative unsquared L2.The authors attribute EMA's performance to smoothly tracking the evolving representation space and providing more stable function-level targets.Source: Figure 6 · p. 7, §4.3 · pp. 7–8

4.2 Main Results

  • On AppWorld, ToolCompass exceeds CORAL by 6.48 OOD points and 6.27 total points; on FTRL, it exceeds GRPO by 3.61 OOD points and ToolRL by 7.12 total points.The compared methods use GRPO with Qwen3.5-4B.Source: §4.2 · p. 6, Table 3 · p. 8

4.3 Analysis

  • ToolCompass makes 26.51 calls versus GRPO's 31.55 and MatchTIR's 20.87, while retaining 12.61 useful trials versus 10.47 and 4.21, respectively.It makes 4.07 unproductive trials versus GRPO's 12.75.Source: Figure 7 · p. 8, §4.3 · p. 8
  • ToolCompass trained on FTRL achieves 60.51% AppWorld task success, below the 66.44% achieved when trained on AppWorld, while outperforming vanilla GRPO in both transfer directions.The authors report that benefits extend to cross-dataset transfer.Source: Table 4 · p. 8, §4.3 · p. 8

5 Related Work

Related work spans general OOD generalization, tool-use agents, and turn-level supervision. ToolCompass instead organizes tool-call representations by shared function during post-training.

OOD generalization

  • Conventional OOD methods use domain alignment, robust optimization, or representation learning for fixed prediction, while tool-use settings shift across tasks, tools, observations, and interactions.Prior tool-specific approaches include synthetic environments, self-verification, interface optimization, open-world retrieval, generalization-aware rewards, and documentation adaptation.Source: §5 · p. 8
  • ToolCompass organizes representations by shared function during post-training, enabling transfer to unseen tools without accessing them or adding test-time adaptation.Unlike the described supervision approaches, it needs no additional trajectory supervision, frozen turn scorers, or inference overhead.Source: §5 · pp. 8–9

Tool-use agents

  • Tool-use post-training optimizes trajectory outcomes or learns self-correction from environment feedback, while turn-level methods match gold traces, estimate turn contributions, or localize irrecoverable actions.ToolMaster requires an additional teacher model to generate trialing trajectories.Source: §5 · pp. 8–9

6 Conclusion

ToolCompass guides tool trialing for OOD tool use by shaping tool-call representations using function-class assignments and a vMF objective. The paper reports consistent gains across benchmarks, model scales, and post-training methods.

6 Conclusion

  • ToolCompass is proposed as a framework for guiding tool trialing to improve OOD tool use.It shapes the tool-call representation space using function-class assignments and a vMF objective.Source: §6 · p. 9
  • The framework transfers experience from seen tools to functionally related unseen tools without ground-truth call traces or unseen-tool access.It adds no inference overhead and can be combined with diverse post-training methods.Source: §6 · p. 9
  • Experiments show consistent tool-use performance improvements across benchmarks, model scales, and post-training methods.The conclusion identifies generalizable tool-use agents as a direction for future research.Source: §6 · p. 9

A.1 Dataset Details

The evaluation uses AppWorld and FTRL, benchmarks with executable tools and verifiable outcomes. Their dataset splits and metrics define the ID and OOD settings used in the study.

A.1 Dataset Details

  • The evaluation covers AppWorld and FTRL, two multi-turn tool-use benchmarks with executable tools and verifiable task outcomes.The paper introduces dataset contents, ID/OOD split construction, and evaluation protocols.Source: §A.1 · p. 12
  • AppWorld's Test-N split is treated as ID and Test-C as OOD because Test-C requires an API from unseen Amazon or Gmail applications.The splits contain 168 ID and 417 OOD tasks; success is the percentage passing all state-based evaluation tests.Source: §A.1 · p. 12
  • FTRL's OOD split holds out five domains while keeping their function classes covered by seen domains; its test set has 168 ID and 32 OOD queries.FTRL reports Solve-P, Solve-R, and their harmonic mean, Solve-F1.Source: §A.1 · p. 12

A.2 Input Prompts

  • The study uses the official task prompts for AppWorld and FTRL.The prompts are shown in Figures A.1 and A.2.Source: §A.2 · p. 12

A.3 Implementation Details

The appendix specifies the interaction and training setup, tool-call representation extraction, benchmark tool annotations, and comparison baselines.

A.3 Implementation Details

  • The setup uses Qwen3.5-4B and Qwen3.5-9B with a shared ReAct scaffold, benchmark-specific interaction protocols, and three-seed runs.AppWorld API specifications are retrieved on demand, while FTRL queries supply fixed candidate tool schemas.Source: §A.3 · p. 12

A.4 Tool-Call Representation Extraction

  • Each tool call is represented by its layer-8 final-token hidden state, projected and normalized separately, including for multiple calls in one code block.The same extraction rule applies to FTRL tool calls.Source: §A.4 · p. 13, Figure A.3 · p. 14

A.5 Domain and Function-Class Annotations

  • A shared domain-annotation prompt yields nine AppWorld application domains and 27 FTRL subject domains; GPT-5.6-sol assigns function-class labels to all 473 AppWorld APIs and 4,545 FTRL tools.The 463 ordinary AppWorld APIs form 14 classes; ten meta APIs are labeled META and excluded from the representation-shaping objective, while FTRL tools form nine classes. Fleiss’s kappa is 0.86 on AppWorld and 0.83 on a random sample of 200 FTRL tools.Source: §A.5 · p. 14

A.6 Baseline Details

  • Baselines span prompting-only models, general post-training, turn-level supervision, tool-use RL, representation learning, and OOD-generalization methods.The appendix lists GPT-5.5, Claude Opus 4.8, GLM-5.2, and DeepSeek V4 Pro among prompting baselines, and describes the distinct training objectives used for comparison.Source: §A.6 · p. 15

A.7 Compute Resources and Time

The appendix reports compute costs, outlines the joint training procedure, and states the default host objective for the additional analysis.

A.7 Compute Resources and Time

  • A complete Qwen3.5-4B ToolCompass run takes about 6.6 hours on AppWorld and 21.9 hours on FTRL, with added training time below 1% versus vanilla GRPO.Experiments use Python 3.12.13 and PyTorch 2.11.0 with CUDA 12.8 on four NVIDIA H200 GPUs.Source: §A.7 · p. 15

A.8 Training Algorithm

  • The training algorithm jointly updates the policy and projection head using the host objective and representation-shaping losses, then updates prototypes with EMA.It extracts and labels observed tool-call representations and computes variation and separation losses from each batch.Source: §A.8 · p. 18, §A.7 · p. 18

B Additional Analysis

B.1 Semantic OOD Analysis

The analysis examines semantic OOD transfer, alternative representation objectives, hyperparameter sensitivity, and qualitative tool-trialing behavior.

B.1 Semantic OOD Analysis

  • ToolCompass improves FTRL semantic-OOD Solve-P and Solve-R over GRPO by 2.18 and 6.12 points, respectively.The evaluation holds out CALCULATE_MEASURE, ANALYZE_SUMMARIZE, and IDENTIFY_CLASSIFY; t-SNE shows more coherent within-function groups and clearer separation under ToolCompass.Source: §B.1 · p. 18, Figure B.1 · p. 19

B.2 Comparison of Auxiliary Objectives

  • ToolCompass outperforms Function CE and SupCon by 7.44 and 3.12 percentage points in AppWorld OOD task success, respectively.The comparison uses the same function labels, representation layer, projection head, normalization, and training budget.Source: §B.2 · p. 18, Table B.2 · p. 19

B.3 Effect of Loss Weights

  • 66.44% task success is reached at λvMF = 0.02 and λsep = 2.0; too little or too much weighting weakens performance.The account links low weights to insufficient supervision or separation, and high weights to overemphasizing shaping or reducing within-class flexibility.Source: §B.3 · p. 19, Figure B.2 · p. 19

B.4 Effect of the EMA Factor

  • 66.44% task success is reached at EMA factor α = 0.9; smaller α fluctuates with minibatches, while larger α adapts too slowly.The tested endpoints are batch-estimated prototypes at α = 0 and frozen initial prototypes at α = 1.0.Source: §B.4 · p. 19, Figure B.3 · p. 20

B.5 Qualitative Analysis

  • Across two AppWorld cases, ToolCompass trials related APIs and corrects errors, while GRPO wastes trials or fails and MatchTIR stops before final actions.The cases require an Amazon checkout and replacing an attachment in a Gmail draft.Source: §B.5 · p. 20

B.6 AppWorld SGC Results

ToolCompass leads on AppWorld Scenario Goal Completion across splits, with gains on OOD scenarios over LOOP and SEAL.

B.6 AppWorld SGC Results

  • ToolCompass achieves the best AppWorld SGC on all splits, improving OOD SGC by 5.28 points over LOOP and 3.12 points over SEAL.SGC credits a scenario only when all its task variants are completed successfully.Source: §B.6 · p. 20

B.7 FTRL Solve-P, Solve-R, and Solve-F1 Results

On FTRL, ToolCompass leads across Solve-P, Solve-R, and Solve-F1 on the ID, OOD, and total test sets. The appendices also describe tool-call annotation and the representation-space visualization.

B.7 FTRL Solve-P, Solve-R, and Solve-F1 Results

  • ToolCompass achieves the best Solve-P, Solve-R, and Solve-F1 results across the ID, OOD, and total FTRL test sets, covering invocation precision and task-completion recall.The comparison uses Qwen3.5-4B and includes LOOP and SEAL.Source: §B.7 · p. 20
  • On the Amazon gift-wrapping task, ToolCompass completes checkout in 29 calls with 5 unproductive ones, while GRPO exhausts its budget and MatchTIR stops before checkout.ToolCompass keeps trials on checkout-relevant APIs, whereas GRPO spends 16 calls on unrelated or repeated queries.Source: Figure B.4 · p. 21
  • On the Gmail attachment task, ToolCompass sends the email in 22 calls with 3 unproductive ones; GRPO also succeeds, while MatchTIR stops without sending.GRPO spends 10 of its 28 calls on unrelated mailbox checks and repeats; ToolCompass uses targeted follow-ups to resolve path and upload-format errors.Source: Figure B.5 · p. 22

B.8 Tool-Call Categorization

  • The tool-call judge labels calls as direct use, useful trialing, or unproductive trialing based on the task, tool specifications, interaction history, current call, and immediate feedback.Novelty is judged against specifications and prior history; later actions and final outcomes are excluded.Source: §B.8 · p. 23
  • Three annotators independently checked 200 calls, yielding Fleiss’s kappa of 0.73 and 82.00% raw agreement; LLM labels matched final reviewed labels on 86.00% of calls.The remaining annotator disagreements were resolved by discussion.Source: §B.8 · p. 23

B.9 Representation-Space Visualization Details

  • Figure 4 visualizes four function classes using normalized layer-8 backbone states without the projection head, mapped onto a unit sphere with shared viewpoint and color scale.The surface represents the spherical kernel density of the displayed class.Source: §B.9 · p. 23
Loading 2609.25678v1…