RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics

• Over 50% of crawled tools have critical flaws, and semantic similarity alone cannot identify which functionally similar tools will execute successfully.Existing approaches such as ToolLLM [21] and ToolExpNet treat retrieval as static ranking, while runtime failures require prompt candidate reordering.Source: §1 · p. 1
• RATED combines a contrastively fine-tuned dual-encoder for semantic matching with a lightweight contextual bandit that re-ranks top-K candidates using metadata and execution feedback.The bandit updates from individual execution outcomes and adjusts ordering between requests without retraining the semantic retriever.Source: §1 · pp. 1–2
• EcoAgent comprises 5,172 real-world agent tools and over 210,000 query–tool pairs for evaluating retrieval and reliability.Source: §1 · p. 1
• Under 30% failure injection, RATED reduces fault selection rate to 2.9%, versus 86.2% for the ToolLLM API retriever, with 0.172 ms latency per step over encode-only inference.Source: §1 · p. 2

2 The EcoAgent Benchmark

• EcoAgent aggregates 5,172 tools from four agent platforms, scores each tool’s metadata reliability, and reserves EcoAgent-Syn for training and EcoAgent-Real for retrieval and dynamic evaluation.Scores combine name–description consistency, source credibility, and documentation quality; 2,940 tools (56.8%) meet the reliable threshold of 0.8.Source: §2 · p. 2
• EcoAgent-Syn uses Gemma-3-1b [7] to create 69,501 queries linked to 5,172 tools, yielding 70,122 positive and 140,244 negative pairs.Queries are task-oriented, deduplicated, and paired with SBERT [22] hard negatives and random negatives.Source: §2 · p. 2
• EcoAgent-Real, built from ToolBench [21], contains 6,379 queries and 9,946 verified query–tool pairs after semi-automated labeling and filtering.GTE-Large [12] retrieves candidates, and Gemma-3-1b [7] verifies functional matches; human experts agreed with the verifier on 94% of a stratified sample of 200 queries.Source: §2 · p. 2

3.1 Problem Setup

• The retrieval problem selects one tool from a top-K candidate set drawn from EcoAgent’s 5,172-tool repository, aiming to maximize execution success under partial feedback.Only the chosen tool returns binary execution feedback, so utility reflects both semantic relevance and time-varying reliability.Source: §3.1 · p. 2

3.2 Semantic Perception Module

• A DRAGON-based semantic module retrieves the top-K tools, fine-tuning its encoders on EcoAgent-Syn with in-batch InfoNCE and hard negatives before freezing them for deployment.New tools can be added by encoding their documentation with the tool encoder, without retraining; semantic relevance uses the dot product of normalized query and tool representations.Source: §3.2 · p. 2, §3.2.1 · p. 2

3.2.1 DRAGON Dual-Encoder Architecture.

• DRAGON uses separate query and tool encoders, whose L2-normalized CLS representations are scored by their dot product.The separate encoders accommodate differences between query language and tool documentation.Source: §3.2.1 · p. 2

3.3 Reliability-Aware Re-ranking

• Discounted LinUCB re-ranks candidates using query context and execution feedback, down-weighting older rewards so recent failures can change tool scores.Each tool has a separate linear model; feedback is binary and observed only after the selected tool executes. The method is designed to address the slow adaptation of standard LinUCB [11] and the lack of query-context modeling in Global MAB, using discounted LinUCB.Source: §3.3 · p. 3
• For cold-start tools, RATED initializes each bandit arm from a static prior based on name-description consistency, source credibility, and documentation quality.These signals use catalog fields without manual ratings; the prior initializes the arm bias when a tool has little or no execution history.Source: §3.3 · p. 3

3.4 Hybrid Fusion for Tool Ranking

• RATED combines semantic relevance, bandit reliability, and static catalog reliability in a weighted score, then executes the highest-ranked candidate and updates the bandit.The weights sum to 1, are tuned on validation data, and are fixed for testing.Source: §3.4 · pp. 3–4

4.1 Experimental Setup

• The experiments fine-tune DRAGON on EcoAgent-Syn and re-rank the top 10 semantic matches using a contextual bandit with diagonal covariance updates.The bandit uses α=0.1, γ=0.95, and 2,000 warm-up pairs; fusion weights are tuned on the EcoAgent-Real validation split.Source: §4.1 · p. 4

4.2 Evaluation Metrics

• Evaluation measures offline MRR, Recall@1, and Recall@5, and averages dynamic reliability metrics over 3,000 post-onset steps with a fixed persistent outage set.The outage set marks the next ρN tools as failing after skipping the 10% most frequent ground-truth tools; dynamic evaluation runs for 10,000 sequential selections.Source: §4.2 · p. 4

4.4 Main Results

• RATED (Full) improves EcoAgent-Real MRR by 52.2% over ToolLLM; under injected failures, its FSR is 2.5% and 2.9% while Post-Fail SR stays above 82%.The reported FSR reductions are 7.3× over pretrained DRAGON and 30× over ToolLLM. Static retrievers do not recover within the evaluation horizon.Source: Table 2 · p. 4, §4.4 · pp. 4–5

4.3 Baselines

• Retrieval comparisons span lexical, dense, neural, cross-encoder, and ToolLLM baselines; dynamic comparisons include blacklisting, retry, batch GRPO, and RATED variants.RATED uses EcoAgent-Syn-fine-tuned DRAGON as its backbone.Source: §4.3 · p. 4

4.5 Hyperparameter Sensitivity

• At 20% failure injection, Post-Fail SR peaks at fusion weights w1=0.75 and w3=0.10; with those weights fixed, it is highest at γ=0.95.The selected bandit setting is α=0.1; FSR increases slowly across the sweeps.Source: Figure 4 · p. 5, §4.5 · p. 5

4.6 Efficiency Analysis

• RATED (Full) adds 0.172 ms over encode-only inference, reaching 3.395 ms average and 4.391 ms P95 latency.The encode-only average is 3.223 ms; batch GRPO similar to ToolRL adds 17.839 ms average.Source: §4.6 · p. 5

5 Conclusion

• RATED combines DRAGON encoders fine-tuned with contrastive learning and contextual-bandit re-ranking driven by execution feedback.The framework is designed for reliable tool retrieval.Source: §5 · p. 5
• Under injected tool failures, RATED reduces fault selection to 2.5–2.9% FSR, 7.3× lower than static DRAGON.The reported added latency is 0.172 ms over encode-only inference.Source: §5 · p. 5
• On EcoAgent, RATED achieves strong in-domain retrieval and remains competitive on real-world queries.It adds 0.172 ms latency over encode-only inference.Source: §5 · p. 5
• Evaluating RATED’s cold-start path on newly added tools and validating it in live agent runtimes remain future work.Source: §5 · p. 5

GenAI Usage Disclosure

• Gemma-3-1b generated EcoAgent-Syn queries and verified candidate query–tool pairs during EcoAgent-Real labeling; ChatGPT improved sentence clarity and grammar.The authors released the prompt templates with the dataset.Source: GenAI Usage Disclosure · p. 5