AgentSearchBench: A Benchmark for AI Agent Search in the Wild

• Agent capabilities are compositional and execution-dependent, so textual descriptions provide only a partial signal of real competence.Source: §1 · p. 1
• Existing benchmarks typically assume structured functionality, controlled candidate pools, or executable queries rather than realistic open-ecosystem search.Source: §1 · p. 2, Table 1 · p. 2
• AgentSearchBench contains nearly 10,000 real-world agents and formalizes search as retrieval and reranking for executable queries and high-level task descriptions.Source: §1 · p. 2
• The benchmark defines relevance using execution-grounded performance signals and converts execution outcomes into fine-grained retrieval and ranking annotations.Source: §1 · p. 2
• Experiments reveal a consistent semantic–performance gap, while lightweight behavioral probing substantially improves ranking quality.Source: §1 · p. 2

Agentic Systems and Orchestration

• Agentic-system research commonly studies collaboration and workflow composition with predefined, limited candidate sets.Source: §2 · pp. 2–3

Information Retrieval and Learning-to-Rank

• Open-ecosystem agent search instead must handle overlapping capabilities and relevance that depends on execution.Source: §2 · p. 3

Tool Retrieval and Selection

• Tool retrieval methods typically use textual descriptions or structured schemas and evaluate executable queries with predefined inputs and outputs.Source: §2 · p. 3

Information Retrieval and Learning-to-Rank

• Information-retrieval and learning-to-rank methods estimate relevance from textual similarity or annotations while assuming relevance is static and observable without interaction.Source: §2 · p. 3

3.1 Agent Search Problem

• Agent search scores candidates against a task specification and produces a ranked list from a large agent repository.Source: §3.1 · p. 3
• The task has two objectives: retrieve top-k agents capable of solving it and rank them by task-performance quality.Source: §3.1 · p. 3

3.2 Task Query and Task Description

• The framework supports executable task queries and high-level task descriptions as distinct levels of task specification.Source: §3.2 · p. 3
• A task query is a concrete executable instruction evaluated by running agents, including single-agent and multi-agent tasks composed of executable subtasks.Source: §3.2 · p. 3
• A task description expresses a high-level goal that is not directly executable, such as modernizing a legacy codebase to current enterprise standards.Source: §3.2 · p. 3
• The benchmark pipeline generates executable queries, derives grounded high-level descriptions, and converts execution outcomes into retrieval and ranking labels.Source: Figure 1 · p. 4

3.3 Task-Performance-Based Relevance

• Relevance is based on execution performance: direct task completion for executable queries and aggregated performance over associated queries for task descriptions.Source: §3.3 · p. 4
• Documentation–performance alignment is an auxiliary ranking-label signal when observed success exceeds documented functionality, because such success may be less reliable.Source: §3.3 · p. 4

Agent Repository

• AgentSearchBench uses nearly 10,000 real-world agents from public platforms to capture capability overlap and inconsistent documentation.The repository is sourced from open agent ecosystems rather than synthesized agents.Source: §4.1 · pp. 4–5

Task Query Construction

• The benchmark generates executable task queries from agent documentation and reduces evaluation cost with hybrid lexical, semantic, and tool-aware retrieval.The hybrid candidate function combines BM25, BGE, and ToolRet scores before execution-based evaluation.Source: §4.2 · p. 5
• Multi-agent queries compose executable subtasks from capability-aligned clusters and are retained only when natural-language inference confirms all subtasks are entailed.This filtering step preserves semantic coverage of the composed query.Source: §4.2 · p. 5

Task Description Construction

• High-level task descriptions are abstracted from semantically related query clusters, with outliers removed and one description generated per cluster.A rubric-based judge then selects the top two queries for each relevance aspect, producing ten queries when D = 5.Source: §4.3 · p. 5

Relevance Annotation

• Agent relevance is grounded in execution using a 5-point LLM judge, with labels supporting both binary retrieval and graded subtask-completion relevance.Agents that succeed without corresponding documented capability receive discounted scores, such as 0.5, to reflect documentation–performance misalignment.Source: §4.4 · p. 5

Benchmark Scale

• 9,760 agents were collected from multiple open platforms, including 7,867 with executable interfaces.The benchmark also includes 2,952 executable task queries and 259 task descriptions.Source: §5 · pp. 5–6
• 66,740 execution runs evaluate the top-20 retrieved agents for the benchmark’s task queries.Each of the 2,952 executable queries is evaluated against its top-20 retrieved agents.Source: §5 · p. 6

Diversity

• Agents and tasks exhibit broad, long-tailed semantic distributions, reflecting diverse capability coverage.These distributions are summarized in Figure 2.Source: §5 · p. 6, Figure 2 · p. 6

Difficulty

• Many queries have multiple relevant agents, while substantial performance variance across agents motivates fine-grained reranking.The benchmark therefore tests more than retrieving a single suitable candidate.Source: §5 · p. 6, Figure 3 · p. 6
• Agents typically cover only subsets of subtasks, highlighting partial and overlapping capabilities.This structure contributes to the benchmark’s realistic evaluation setting.Source: §5 · p. 6

6.1 Experimental Setup

• Retrieval evaluates full-repository search using binary execution-based relevance, while reranking orders the top-20 candidates against execution-derived golden rankings.Retrieval uses Precision, Recall, NDCG, and Completeness; reranking uses graded NDCG and Completeness.Source: §6.1 · p. 6

6.2 Benchmarking Analysis

• On task queries, tool-aware retrievers outperform dense and sparse baselines, while BGE achieves the strongest overall performance on task descriptions.These comparisons are reported under execution-based relevance.Source: §6.2 · p. 7
• Performance drops substantially on high-level task descriptions, and low completeness shows that retrieval struggles to find agents satisfying abstract requirements comprehensively.Retrieval captures coarse relevance but has difficulty identifying comprehensive task-solving capability without explicit executable demands.Source: §6.2 · p. 8
• Reranker families perform similarly on concrete task queries, whereas decoder-only and LLM-based rerankers perform more strongly on high-level task descriptions.Improved ordering nevertheless leaves completeness limited for complex requirements.Source: §6.2 · p. 8
• All model rankings remain substantially below the oracle, indicating that many high-performing agents are ranked too low because documentation misaligns with execution performance.Accumulated golden performance gains are distributed gradually rather than concentrated at top ranks.Source: §6.2 · p. 8

6.3 Benchmark Validation

• Synthetic and realistic queries preserve relative method trends, but realistic queries have lower absolute performance because highly relevant agents are not guaranteed in the candidate pool.Dense and tool-aware methods remain stronger than sparse retrieval across settings.Source: §6.3 · p. 8
• LLM-based relevance labels agree strongly with human judgments, reaching Cohen’s kappa κ = 0.93 and accuracy 96.67% across 500 execution instances.Three AI PhD-level annotators participated in the human evaluation.Source: §6.3 · p. 8
• Most methods improve with full-document indexing and execution-aware probing, with probing most effective when responses show medium or high variance across agents.These behavioral signals complement description-based ranking by providing evidence beyond static documentation.Source: §6.4 · p. 9

7 Conclusion

• AgentSearchBench covers nearly 10,000 real-world agents and supports executable task queries and high-level task descriptions.The benchmark grounds relevance in execution outcomes.Source: §7 · p. 9
• Textual-similarity methods often fail to identify the best-performing agents, especially for abstract and multi-step tasks.Existing retrieval and reranking approaches provide useful coarse signals but remain limited in capturing execution-dependent capability.Source: §7 · p. 9
• Richer indexing and execution-aware probing can improve ranking quality by incorporating lightweight behavioral signals.The benchmark is presented as a practical testbed for performance-grounded agent search.Source: §7 · pp. 9–10

A.2 Example of Task Query and Task Description

• Table 6 illustrates examples of single-agent task queries, multi-agent task queries, and task descriptions.Source: §A.2 · p. 15

A.3 Implementation Details of Benchmark Construction

• Task descriptions are generated from 100 candidate queries and filtered to 10 using five rubric criteria, with approximately two subtasks per criterion.GPT-5.2 is used for generation with temperature τ = 1.Source: §A.3 · p. 15
• Multi-agent task queries sample 2–4 semantically related but non-redundant subtasks, with a mean of 2.91.Candidates are retrieved using top-K similarity to an anchor task query.Source: §A.3 · p. 15
• Candidate agents are retrieved with a hybrid score combining BM25, BGE, and ToolRet as lexical, semantic, and tool-aware components.The component weights satisfy α + β + γ = 1, and each score is min-max normalized before top-K selection.Source: §A.3 · p. 15

B.1 Completeness Computation

• Completeness requires the top-K retrieved tools to include at least one relevant tool for every subtask.For single-agent queries, this reduces to standard hit rate.Source: §B.1 · p. 15

Benchmark Examples and Evaluation

• The setup includes examples of unified agent representations and single-agent, multi-agent, and task-description queries, plus evaluations on HLE and Finance Agent Benchmark tasks.The cited passages identify the example tables and the two external realistic benchmarks used for results.Source: Table 4 · p. 16, Table 5 · p. 16, Table 6 · p. 17

Retrieval

• The benchmark evaluates sparse, dense, and tool-specific retrieval model families.The supplied passage introduces these retrieval categories and their cited representative approaches.Source: §B.2 · p. 16

Reranking

• Reranking experiments cover cross-encoder, tool-specific, decoder-only, and LLM-based model families.These families differ in how they score query–document relevance, including joint encoding, autoregressive scoring, and prompted judgments.Source: §B.2 · p. 16

Execution-Aware Probing

• Execution-aware probing uses an assistant to score multiple agent responses on a 1–5 response-quality scale and output ordered scores.The rubric ranges from fully correct and sufficient responses to incorrect, irrelevant, or task-failing responses.Source: §B.3 · p. 17, —SYSTEM— · p. 17

Task Description

• Table 7 reports retrieval results on task descriptions with full indexing.The supplied passage identifies the evaluation setting but does not provide the result values.Source: Table 7 · p. 18

Single-Agent Task Query

• Table 8 reports retrieval results on single-agent task queries with full indexing.The supplied passage identifies the table’s setting but does not provide the result values.Source: Table 8 · p. 18

Multi-Agent Task Query

• Table 9 reports retrieval results on multi-agent task queries with full indexing.The supplied passage identifies the table’s setting but does not provide the result values.Source: Table 9 · p. 18

Task Query Results

• Table 10 reports retrieval results on 200 real task queries.The passage does not specify the numerical outcomes or evaluated methods.Source: Table 10 · p. 19

Task Description Reranking

• Table 11 reports reranking results on task descriptions using golden labels and full indexing.The supplied passage identifies the evaluation configuration but does not provide numerical results.Source: Table 11 · p. 19

Task Query Reranking

• Table 12 reports reranking results for single-agent and multi-agent task queries with golden labels and full indexing.The supplied passage identifies the task-query settings but does not provide numerical outcomes.Source: Table 12 · p. 19