Source-linked AI summary
Multi-Tool Selection for Robotic Systems: Comparing Semantic Retrieval and LLM
Miguel Sousa, Luis Ramos Pinto, João P. Matos-Carvalho
TL;DR
Routing natural-language requests to the complete, minimal set of robot capabilities is difficult for compound instructions, where requests may require multiple tools. The study compares fixed-threshold, relative-distance, and clause-aware ChromaDB retrieval with direct selection by a cloud-hosted 120B LLM. The direct LLM baseline achieved macro-average score 0.8869, exceeding CDBv3's 0.7822, while CDBv3 runs locally with substantially lower and less variable end-to-end routing latency.
Problem
Method
The study compares fixed-threshold, relative-distance, and clause-aware ChromaDB retrieval with direct selection by a cloud-hosted 120B LLM.
Source: §1 · p. 3Results
The direct LLM baseline achieved macro-average score 0.8869, exceeding CDBv3's 0.7822, while CDBv3 runs locally with substantially lower and less variable end-to-end routing latency.
Source: §6 · p. 13Takeaways & Limitations
The preferred approach depends on deployment priorities: large-model routing favors selection quality, while CDBv3 suits local execution, predictable response time, privacy, or connectivity independence.
Source: §6 · p. 13CDBv1–CDBv3 hyperparameters were manually tuned on the same 164-prompt benchmark used for evaluation, so their scores are best-case estimates rather than independent-test results.
Source: §6 · p. 13
Abstract
from arXiv · showhide
Natural-language interfaces allow users to command robotic systems without interacting directly with mission planners or robotics middleware. Their reliable use depends on robot tool routing, defined here as the selection of the smallest complete subset of predefined executable capabilities required to satisfy a request. This task is particularly difficult for compound instructions, where incomplete selection may omit necessary sensing or monitoring capabilities, while excessive selection may expose irrelevant tools. We compare four routing strategies: three progressively refined ChromaDB-based methods - fixed-threshold retrieval, relative-distance retrieval, and clause-aware aggregated retrieval - and a cloud-hosted 120-billion-parameter language-model baseline. The methods are evaluated on 164 manually annotated prompts organised into three complexity levels, from direct single-tool requests to compound requests requiring at least three tools. Using exact-match accuracy, completeness, precision, a completeness-weighted composite score, and routing latency, CDB-v3 achieves the best macro-average score among the retrieval-only methods, with 0.7822, compared with 0.8869 for the large-model baseline. Although the LLM provides higher routing quality, CDB-v3 exhibits substantially lower latency and can be deployed locally without requiring cloud connectivity. These results identify a quality-latency trade-off between direct large-model selection and clause-aware semantic retrieval for privacy-sensitive, connectivity-constrained, and resource-constrained robotic systems.
1 Introduction
The paper frames robotic tool routing as selecting the required capabilities from natural-language requests, including compound requests that need coordinated tools. It compares three semantic-retrieval strategies with a cloud-hosted LLM and finds a quality–latency trade-off.
1 Introduction
- Reliable routing must select all capabilities needed by compound requests, such as both telemetry and actuation for monitoring a battery and landing below a threshold.Missing a required capability may prevent execution, while extra selections expose irrelevant tools.Source: §1 · p. 2
- Cloud-hosted LLM routing may be impractical where connectivity, confidentiality, bandwidth, latency, or edge resources constrain deployment.These conditions motivate local routing approaches that aim to preserve selection quality.Source: §1 · p. 2
- The study compares fixed-threshold, relative-distance, and clause-aware retrieval against direct selection by a cloud-hosted 120-billion-parameter LLM.The evaluation uses 164 manually annotated requests across three complexity levels, with shared ground-truth assignments and measures of exact match, completeness, precision, composite score, and latency.Source: §1 · p. 2
- CDBv3 scored 0.7822 on the macro-average composite score, ahead of CDBv2 at 0.5863 and CDBv1 at 0.3330, while the 120B LLM scored 0.8869.CDBv3 also provides substantially lower latency and does not require cloud connectivity.Source: §1 · p. 3
2 Related Work
Prior work covers tool use, planning, code generation, and retrieval, but this paper focuses on selecting predefined robot capabilities rather than generating executable code or orchestrating tasks. It addresses limited empirical comparison of embedding-only routing policies with direct large-model selection under a shared catalog and evaluation.
2 Related Work
- Tool-using language-model work studies reasoning-and-action cycles, learned tool-call decisions, and instructions over large API collections.These works concern broader tool use than the controlled robot-routing comparison here.Source: §2 · p. 3
- Unlike task-planning, policy-generation, and orchestration systems, this study evaluates routing among independently executable, allowlisted robot tools.The router selects known capability definitions rather than creating arbitrary commands or shell instructions.Source: §2 · p. 3
- Dense retrieval matches requests to capability descriptions in a shared embedding space, and ChromaDB supplies storage, indexing, and similarity search rather than a new routing algorithm [5, 10].The evaluated contribution is in the routing policies built on that infrastructure.Source: §2 · pp. 3–4
- Software-engineering retrieval systems connect models to code or validated function blocks, whereas this work tests semantic retrieval alone for selecting predefined robot processes.It does not generate control code.Source: §2 · p. 4
- Related agent architectures separate capabilities, routing, and execution, but their evaluations target software repositories or broader agent systems rather than real-time robotic capability selection,,.Prior work also reports integration defects in retrieval-supported LLM applications.Source: §2 · p. 4
- The identified gap is limited empirical comparison of embedding-only retrieval policies with direct large-model selection using the same capability catalog, annotations, and metrics.The paper compares three progressively refined retrieval methods with a 120B LLM, considering routing quality and latency.Source: §2 · pp. 4–5
3 Problem Formulation
The formulation treats routing as multi-label selection from a finite catalog of predefined tools. It evaluates whether selections are complete and minimal, while leaving downstream execution and coordination outside scope.
3 Problem Formulation
- The router selects a subset of a finite catalog whose tools have predefined executable interfaces; it cannot create new tools or arbitrary commands.A request may map to no tool, one tool, or several tools.Source: §3 · p. 5
- For compound requests, selecting only the landing tool is incomplete when battery and wind observations are needed to evaluate the triggering conditions.Selecting an extra, unnecessary tool can be complete but less minimal.Source: §3 · p. 5
- The evaluation uses exact-match accuracy, completeness, and precision, macro-averaged over requests.Completeness is weighted more heavily in the composite because omitting a required sensing or action tool may prevent execution; exact match separately rewards predictions that are complete and minimal.Source: §3 · pp. 5–6
- The formulation evaluates tool selection only, not tool arguments, execution order, control flow, or preservation of logical and temporal dependencies.Those concerns are assigned to downstream planning and orchestration.Source: §3 · p. 6
4 Routing Strategies
The section compares three ChromaDB retrieval policies with a direct 120B LLM baseline, all selecting tools from the same catalog. The retrieval policies vary how they match requests to capabilities, with CDBv3 additionally decomposing compound requests into fragments.
4 Routing Strategies
- All four routers select from the same tool catalog; CDBv1, CDBv2, and CDBv3 use retrieval, while the fourth method is a direct large-model baseline.Each method receives the same natural-language request.Source: §4 · p. 6
CDBv1: Fixed-Threshold Retrieval
- CDBv1 embeds tool representations at startup and queries them with the embedded full request, but one threshold can omit required tools or return unnecessary ones.A strict threshold may miss tools for compound requests, while a permissive threshold may include semantically related but unnecessary tools.Source: CDBv1: Fixed-Threshold Retrieval · p. 7
CDBv2: Relative-Distance Retrieval
- CDBv2 selects additional tools relative to the nearest result, adapting candidate-set size to local distances while still embedding each request as one undivided sequence.It retains the nearest tool; similarly close results can yield several candidates, while a clearly separated nearest result yields a smaller set.Source: CDBv2: Relative-Distance Retrieval · p. 7
CDBv3: Clause-Aware Aggregated Retrieval
- CDBv3 queries both the full prompt and deduplicated intent-like fragments, aggregates tool evidence across searches, adds lexical matches, and truncates the ranked list.Fragments may be split around punctuation and coordinating, conditional, or temporal terms; output is capped at eight tools, and no language model performs a second selection.Source: CDBv3: Clause-Aware Aggregated Retrieval · pp. 7–8
Direct 120B LLM Baseline
- The direct 120B LLM receives the complete catalog and request and returns the smallest complete tool set, but depends on cloud connectivity and remote inference latency.It can explicitly interpret logical operators, monitored conditions, dependencies, and compound instructions; structured-output reliability is also a dependency.Source: Direct 120B LLM Baseline · p. 8
5 Routing Results
Across the evaluated request types, CDBv3 is the strongest retrieval-only router, but the direct 120B LLM retains the highest routing score. CDBv3 is faster and connectivity-independent, revealing a deployment trade-off rather than one universally dominant method.
5.1 Experimental Setup
- The 164-request benchmark spans three complexity levels, and CDB hyperparameters were tuned on that same benchmark, making reported scores best-case upper-bound estimates.The tool catalog and ground-truth assignments were prepared by a robotics domain expert.Source: §5.1 · p. 8
5.2 CDBv1, CDBv2 and CDBv3
- Clause-aware retrieval improves completeness for two-tool requests and accuracy for multi-tool requests, though additional related tools can also produce false positives.CDBv1 retains most required tools but admits many unrelated ones; CDBv2 improves direct single-tool routing but remains limited on compound requests.Source: Figure 1 · p. 9, §5.2 · p. 9
- CDBv3 has the highest composite score among retrieval methods by combining high completeness with improved exact-match accuracy and precision, particularly on compound requests.CDBv1's permissive selections often include unnecessary tools and yield poor precision.Source: §5.2 · p. 9, Figure 2 · p. 10
5.3 CDBv3 versus the 120B LLM Baseline
- The direct LLM achieves higher exact-match accuracy across request types, while CDBv3 has similar or higher completeness; the LLM more often removes unnecessary related tools.The quality difference is more relevant when several conditions or actions must be interpreted jointly.Source: Figure 3 · p. 10, §5.3 · pp. 10–11, Figure 4 · p. 11
- CDBv3 operates on the order of milliseconds, while the cloud baseline operates on the order of seconds, with larger latency outliers.The end-to-end comparison includes network conditions, cloud-server availability, and inference queuing.Source: Figure 5 · p. 11, §5.3 · p. 12
5.4 Composite Score
- CDBv3 leads retrieval-only routers for all request types, while the direct LLM exceeds its macro-average composite score by approximately 0.105.CDBv3 improves the macro-average by approximately 0.196 over CDBv2.Source: Table 1 · p. 12, §5.4 · p. 12
6 Conclusion
CDBv3 is the strongest retrieval-only router in the benchmark, while the cloud-hosted LLM achieves higher routing quality. The choice depends on deployment priorities, and the retrieval scores are best-case estimates because the parameters were tuned on the evaluation benchmark.
6 Conclusion
- The benchmark compared three ChromaDB routers with a direct cloud-hosted LLM on 164 manually annotated requests across three complexity categories.The categories cover single-tool, exactly two-tool, and three-or-more-tool requests.Source: §6 · p. 12
- CDBv3 scored 0.7822, versus 0.5863 for CDBv2 and 0.3330 for CDBv1; the 120B LLM scored 0.8869.Clause-aware retrieval was strongest among retrieval-only methods, but did not exceed the direct LLM baseline.Source: §6 · pp. 12–13
- CDBv3 offers lower and less variable latency with local execution, while the cloud baseline depends on cloud inference and network conditions.The preferred strategy depends on whether routing quality or local, predictable, connectivity-independent operation is the priority.Source: §6 · p. 13
- CDBv1–CDBv3 hyperparameters were manually tuned on the same 164-prompt benchmark used for evaluation, making their reported scores best-case estimates rather than independent-test results.The paper identifies held-out development data, larger and evolving catalogs, robustness, and execution-aware metrics as future-work areas.Source: §6 · p. 13