PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSearchBench: A Benchmark for AI Agent Search in the Wild

Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine Yilmaz

arXiv:2604.22436v1cs.AIcs.IRcs.MA

TL;DR

AgentSearchBench addresses the challenge of searching for capable agents when compositional, execution-dependent abilities are poorly captured by descriptions and existing benchmarks use restricted settings. It builds a large-scale benchmark that supports executable and high-level task specifications with execution-grounded relevance. Experiments find a semantic–performance gap and show that lightweight behavioral signals improve ranking quality.

  • Problem

    Existing agent-search research and benchmarks inadequately represent open ecosystems because they rely on well-specified functionality, controlled candidate pools, or executable queries.

  • Method

    AgentSearchBench formalizes agent search as retrieval and reranking over nearly 10,000 real-world agents, evaluating relevance through execution-grounded performance for executable and high-level tasks.

  • Results

    Experiments consistently reveal a semantic–performance gap, while lightweight behavioral signals such as execution-aware probing improve ranking quality.

  • Takeaways & Limitations

    Realistic agent discovery benefits from incorporating execution signals because description-based methods can miss high-performing agents.

  • Takeaways & Limitations

    Observed success may be less reliable when it is inconsistent with an agent’s documented functionality, so ranking labels also use documentation–performance alignment.

Abstract

from arXiv · show

The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at https://github.com/Bingo-W/AgentSearchBench.

1 Introduction

AgentSearchBench addresses the difficulty of finding capable agents in open ecosystems, where textual descriptions incompletely predict execution performance and existing benchmarks use unrealistic assumptions. It introduces an execution-grounded benchmark and shows that lightweight behavioral probing improves agent ranking.

  • Agent capabilities are compositional and execution-dependent, so textual descriptions provide only a partial signal of real competence.
  • Existing benchmarks typically assume structured functionality, controlled candidate pools, or executable queries rather than realistic open-ecosystem search.
  • AgentSearchBench contains nearly 10,000 real-world agents and formalizes search as retrieval and reranking for executable queries and high-level task descriptions.
  • The benchmark defines relevance using execution-grounded performance signals and converts execution outcomes into fine-grained retrieval and ranking annotations.
  • Experiments reveal a consistent semantic–performance gap, while lightweight behavioral probing substantially improves ranking quality.

2 Related Work

Related work studies agent orchestration, tool retrieval, and information retrieval, but generally assumes limited candidates, textual or structured matching, and static relevance. AgentSearchBench targets open-ecosystem search in which relevance depends on execution.

  • Agentic Systems and Orchestration: Agentic-system research commonly studies collaboration and workflow composition with predefined, limited candidate sets.
  • Information Retrieval and Learning-to-Rank: Open-ecosystem agent search instead must handle overlapping capabilities and relevance that depends on execution.
  • Tool Retrieval and Selection: Tool retrieval methods typically use textual descriptions or structured schemas and evaluate executable queries with predefined inputs and outputs.
  • Information Retrieval and Learning-to-Rank: Information-retrieval and learning-to-rank methods estimate relevance from textual similarity or annotations while assuming relevance is static and observable without interaction.

3 Problem Formulation

The paper formulates agent search as retrieving and ranking candidates under executable or high-level task specifications. It defines relevance from execution performance, aggregates high-level-task evidence across queries, and supplements success with documentation–performance alignment.

  • 3.1 Agent Search Problem: Agent search scores candidates against a task specification and produces a ranked list from a large agent repository.
  • 3.1 Agent Search Problem: The task has two objectives: retrieve top-k agents capable of solving it and rank them by task-performance quality.
  • 3.2 Task Query and Task Description: The framework supports executable task queries and high-level task descriptions as distinct levels of task specification.
  • 3.2 Task Query and Task Description: A task query is a concrete executable instruction evaluated by running agents, including single-agent and multi-agent tasks composed of executable subtasks.
  • 3.2 Task Query and Task Description: A task description expresses a high-level goal that is not directly executable, such as modernizing a legacy codebase to current enterprise standards.
  • 3.2 Task Query and Task Description: The benchmark pipeline generates executable queries, derives grounded high-level descriptions, and converts execution outcomes into retrieval and ranking labels.
  • 3.3 Task-Performance-Based Relevance: Relevance is based on execution performance: direct task completion for executable queries and aggregated performance over associated queries for task descriptions.
  • 3.3 Task-Performance-Based Relevance: Documentation–performance alignment is an auxiliary ranking-label signal when observed success exceeds documented functionality, because such success may be less reliable.

4 AgentSearchBench: A Benchmark for Agent Search

AgentSearchBench constructs a realistic benchmark from real-world agents and evaluates agent search through execution-grounded retrieval and ranking. Its hierarchical pipeline generates executable queries, abstracts high-level descriptions, and derives relevance labels from execution performance.

  • Agent Repository: AgentSearchBench uses nearly 10,000 real-world agents from public platforms to capture capability overlap and inconsistent documentation.The repository is sourced from open agent ecosystems rather than synthesized agents.
  • Task Query Construction: The benchmark generates executable task queries from agent documentation and reduces evaluation cost with hybrid lexical, semantic, and tool-aware retrieval.The hybrid candidate function combines BM25, BGE, and ToolRet scores before execution-based evaluation.
  • Task Query Construction: Multi-agent queries compose executable subtasks from capability-aligned clusters and are retained only when natural-language inference confirms all subtasks are entailed.This filtering step preserves semantic coverage of the composed query.
  • Task Description Construction: High-level task descriptions are abstracted from semantically related query clusters, with outliers removed and one description generated per cluster.A rubric-based judge then selects the top two queries for each relevance aspect, producing ten queries when D = 5.
  • Relevance Annotation: Agent relevance is grounded in execution using a 5-point LLM judge, with labels supporting both binary retrieval and graded subtask-completion relevance.Agents that succeed without corresponding documented capability receive discounted scores, such as 0.5, to reflect documentation–performance misalignment.

5 Benchmark Statistic

AgentSearchBench is large and execution-centric, covering thousands of agents, executable queries, and high-level task descriptions. Its task and agent distributions are broad and long-tailed, while partial capabilities and performance variation make reranking necessary.

  • Benchmark Scale: 9,760 agents were collected from multiple open platforms, including 7,867 with executable interfaces.The benchmark also includes 2,952 executable task queries and 259 task descriptions.
  • Benchmark Scale: 66,740 execution runs evaluate the top-20 retrieved agents for the benchmark’s task queries.Each of the 2,952 executable queries is evaluated against its top-20 retrieved agents.
  • Diversity: Agents and tasks exhibit broad, long-tailed semantic distributions, reflecting diverse capability coverage.These distributions are summarized in Figure 2.
  • Difficulty: Many queries have multiple relevant agents, while substantial performance variance across agents motivates fine-grained reranking.The benchmark therefore tests more than retrieving a single suitable candidate.
  • Difficulty: Agents typically cover only subsets of subtasks, highlighting partial and overlapping capabilities.This structure contributes to the benchmark’s realistic evaluation setting.

6 Agent Search Evaluation

The benchmark evaluates retrieval and reranking under executable queries and high-level task descriptions using execution-grounded relevance. Results show persistent gaps between textual matching and execution performance, while richer indexing and behavioral probing improve ranking.

  • 6.1 Experimental Setup: Retrieval evaluates full-repository search using binary execution-based relevance, while reranking orders the top-20 candidates against execution-derived golden rankings.Retrieval uses Precision, Recall, NDCG, and Completeness; reranking uses graded NDCG and Completeness.
  • 6.2 Benchmarking Analysis: On task queries, tool-aware retrievers outperform dense and sparse baselines, while BGE achieves the strongest overall performance on task descriptions.These comparisons are reported under execution-based relevance.
  • 6.2 Benchmarking Analysis: Performance drops substantially on high-level task descriptions, and low completeness shows that retrieval struggles to find agents satisfying abstract requirements comprehensively.Retrieval captures coarse relevance but has difficulty identifying comprehensive task-solving capability without explicit executable demands.
  • 6.2 Benchmarking Analysis: Reranker families perform similarly on concrete task queries, whereas decoder-only and LLM-based rerankers perform more strongly on high-level task descriptions.Improved ordering nevertheless leaves completeness limited for complex requirements.
  • 6.2 Benchmarking Analysis: All model rankings remain substantially below the oracle, indicating that many high-performing agents are ranked too low because documentation misaligns with execution performance.Accumulated golden performance gains are distributed gradually rather than concentrated at top ranks.
  • 6.3 Benchmark Validation: Synthetic and realistic queries preserve relative method trends, but realistic queries have lower absolute performance because highly relevant agents are not guaranteed in the candidate pool.Dense and tool-aware methods remain stronger than sparse retrieval across settings.
  • 6.3 Benchmark Validation: LLM-based relevance labels agree strongly with human judgments, reaching Cohen’s kappa κ = 0.93 and accuracy 96.67% across 500 execution instances.Three AI PhD-level annotators participated in the human evaluation.
  • 6.3 Benchmark Validation: Most methods improve with full-document indexing and execution-aware probing, with probing most effective when responses show medium or high variance across agents.These behavioral signals complement description-based ranking by providing evidence beyond static documentation.

7 Conclusion

AgentSearchBench provides a performance-grounded testbed for agent search across nearly 10,000 real-world agents and both executable and high-level tasks. Its findings show that execution-aware signals are important because textual similarity alone often misses high-performing agents.

  • 7 Conclusion: AgentSearchBench covers nearly 10,000 real-world agents and supports executable task queries and high-level task descriptions.The benchmark grounds relevance in execution outcomes.
  • 7 Conclusion: Textual-similarity methods often fail to identify the best-performing agents, especially for abstract and multi-step tasks.Existing retrieval and reranking approaches provide useful coarse signals but remain limited in capturing execution-dependent capability.
  • 7 Conclusion: Richer indexing and execution-aware probing can improve ranking quality by incorporating lightweight behavioral signals.The benchmark is presented as a practical testbed for performance-grounded agent search.

A.2 Example of Task Query and Task Description

This appendix section points to an example table containing single-agent queries, multi-agent queries, and task descriptions.

  • A.2 Example of Task Query and Task Description: Table 6 illustrates examples of single-agent task queries, multi-agent task queries, and task descriptions.

A.3 Implementation Details of Benchmark Construction

The benchmark construction uses generated and filtered task descriptions, sampled related subtasks, and hybrid retrieval to form candidate agent sets. Scores are normalized before top-K selection.

  • A.3 Implementation Details of Benchmark Construction: Task descriptions are generated from 100 candidate queries and filtered to 10 using five rubric criteria, with approximately two subtasks per criterion.GPT-5.2 is used for generation with temperature τ = 1.
  • A.3 Implementation Details of Benchmark Construction: Multi-agent task queries sample 2–4 semantically related but non-redundant subtasks, with a mean of 2.91.Candidates are retrieved using top-K similarity to an anchor task query.
  • A.3 Implementation Details of Benchmark Construction: Candidate agents are retrieved with a hybrid score combining BM25, BGE, and ToolRet as lexical, semantic, and tool-aware components.The component weights satisfy α + β + γ = 1, and each score is min-max normalized before top-K selection.

B More Details about Experimental Setup

The experimental setup defines execution-grounded completeness and surveys retrieval, reranking, and execution-aware probing components. It also includes benchmark task examples and evaluations on external realistic benchmarks.

  • B.1 Completeness Computation: Completeness requires the top-K retrieved tools to include at least one relevant tool for every subtask.For single-agent queries, this reduces to standard hit rate.
  • Benchmark Examples and Evaluation: The setup includes examples of unified agent representations and single-agent, multi-agent, and task-description queries, plus evaluations on HLE and Finance Agent Benchmark tasks.The cited passages identify the example tables and the two external realistic benchmarks used for results.
  • Retrieval: The benchmark evaluates sparse, dense, and tool-specific retrieval model families.The supplied passage introduces these retrieval categories and their cited representative approaches.
  • Reranking: Reranking experiments cover cross-encoder, tool-specific, decoder-only, and LLM-based model families.These families differ in how they score query–document relevance, including joint encoding, autoregressive scoring, and prompted judgments.
  • Execution-Aware Probing: Execution-aware probing uses an assistant to score multiple agent responses on a 1–5 response-quality scale and output ordered scores.The rubric ranges from fully correct and sufficient responses to incorrect, irrelevant, or task-failing responses.

C.2 More Results on Indexing

This section presents additional retrieval and reranking results under full indexing for task descriptions and both single-agent and multi-agent task queries.

  • Task Description: Table 7 reports retrieval results on task descriptions with full indexing.The supplied passage identifies the evaluation setting but does not provide the result values.
  • Single-Agent Task Query: Table 8 reports retrieval results on single-agent task queries with full indexing.The supplied passage identifies the table’s setting but does not provide the result values.
  • Multi-Agent Task Query: Table 9 reports retrieval results on multi-agent task queries with full indexing.The supplied passage identifies the table’s setting but does not provide the result values.
  • Task Query Results: Table 10 reports retrieval results on 200 real task queries.The passage does not specify the numerical outcomes or evaluated methods.
  • Task Description Reranking: Table 11 reports reranking results on task descriptions using golden labels and full indexing.The supplied passage identifies the evaluation configuration but does not provide numerical results.
  • Task Query Reranking: Table 12 reports reranking results for single-agent and multi-agent task queries with golden labels and full indexing.The supplied passage identifies the task-query settings but does not provide numerical outcomes.
Loading 2604.22436v1…