Source-linked AI summary
Overview of the TREC 2025 Million Large Language Models track
Evangelos Kanoulas, Panagiotis Eustratiadis, Jamie Callan, Mark Sanderson, Yongkang Li, Jingfen Qiao, Gabrielle Poerwawinata, Vaishali Pal
TL;DR
Selecting suitable expert LLMs is difficult when expertise is diverse and cannot be represented adequately by static descriptions. The Million LLM Track ranks models from behavioral evidence, and participating systems achieved nDCG@10 scores between 0.15 and 0.17, with retrieval-style profile methods outperforming purely parametric classifiers across the reported variants.
Problem
An assistant must identify relevant expert LLMs for a query without querying every available model, which is costly and inefficient.
Source: §1 · p. 2Method
Results
Retrieval-style profile methods in embedding space were more effective than purely parametric classifiers across the reported variants.
Source: §5 · p. 8Takeaways & Limitations
The track provides a large-scale benchmark for ranking expert models by generalizing from observable behavior to latent expertise.
Source: §1 · p. 2The simulated-domain approach requires a source dataset large enough for thousands of meaningful subcollections and queries with gold answers or relevant documents.
Source: Constructing Thousands of Expert Domains · p. 3
Abstract
from arXiv · showhide
Agentic AI envisions ecosystems of intelligent agents collaboratively solving complex tasks with minimal human intervention. In such ecosystems, each agent possesses specialized expertise, making effective expert selection central to overall system performance. While most current approaches assume a small number of well-documented models, real-world expertise is far more diverse and cannot be adequately captured through static metadata or hand-written descriptions. We anticipate a future with millions of specialized language models (LLMs), each excelling in different domains or problem types. Rather than relying on predefined capability statements, we propose a retrieval-based paradigm in which an assistant agent infers expertise dynamically by examining models' observable behavior. Upon receiving a user query, the assistant ranks candidate LLMs based on demonstrated competence, enabling efficient and adaptive expert selection. The TREC Million LLM Track operationalizes this paradigm by shifting the retrieval target from documents to expert LLMs. Participants are given a discovery set consisting of queries, answers, and log-probabilities from more than one thousand LLMs and are challenged to infer meaningful expertise representations for each model. Given an unseen test query, systems must then rank the LLMs according to their expected performance, providing the first large-scale benchmark for expertise retrieval in agentic AI.
1 Introduction
As information increasingly comes through LLMs, users may need to combine answers from many specialized models. The Million LLM Track reframes retrieval as selecting expert models from behavioral evidence rather than fixed descriptions.
1 Introduction
- The shift toward specialized LLMs includes models trained on proprietary material and models accessing distinct databases.Examples named include BloombergGPT and LLMs developed by Thomson Reuters and Elsevier.Source: §1 · pp. 1–2
- An orchestration agent must select relevant expert LLMs because querying all available models is costly and inefficient.The selection should balance accuracy with cost efficiency while minimizing redundant computation.Source: §1 · p. 2
- The track ranks expert LLMs for unseen queries using their observable behavior, not stated capabilities or fixed metadata.It reframes retrieval from mapping queries to documents to mapping queries to models.Source: §1 · p. 2
- The discovery set provides hundreds of queries paired with each model’s generated answers and token-level log-probabilities for inferring expertise.Systems use this behavioral evidence to rank models by expected competence on an unseen test query.Source: §1 · p. 2
2 Task Description
The task evaluates whether systems can rank language models by expected answer quality for a query. Participants infer expertise from provided model responses and log-probabilities, then rank models for unseen queries.
2 Task Description
- The task ranks a fixed pool of more than one thousand LMs by their expected performance on each query, rather than ranking documents or passages.The goal is to predict which models are most capable of generating a high-quality answer.Source: §2 · p. 2
- Participants infer model expertise from discovery responses, including generated text and token-level log-probabilities, and development queries with ground-truth expertise labels.The development labels identify LMs that produced the best answers for specific queries.Source: §2 · p. 2
- Using the provided resources without new LM queries during evaluation, systems rank all model identifiers for each unseen test query.Rankings are ordered from most to least likely to produce a high-quality answer.Source: §2 · p. 2
- Submissions are evaluated against hidden queries and expertise labels derived from human and LM judgments using nDCG@10 and MRR.The metrics capture ranking accuracy and ordering quality.Source: §2 · p. 2
- The formulation benchmarks generalization from observable model behavior to latent expertise in large-scale expertise retrieval.The paper presents it as the first large-scale benchmark for expertise retrieval in agentic AI.Source: §2 · p. 2
3 Simulating Thousands of Expert LLMs
Because thousands of real domain-specialized LLMs are not yet available, the paper simulates experts with a shared LLM whose retrieval is restricted to domain-specific collections. It then addresses leakage, domain construction, and scalable retrieval.
3 Simulating Thousands of Expert LLMs
- Each simulated expert combines a shared LLM and retriever with a distinct document collection that constrains its domain expertise.A cinema collection, for example, makes the shared LLM behave as a cinema expert; Figure 1 depicts this setup.Source: §3 · p. 3, Figure 1 · p. 3
Preventing Knowledge Leakage
- Complete separation of generation ability from pretrained knowledge is impossible, so the simulation uses prompt instructions and query filtering to mitigate leakage.Filtering removes questions the base model answers correctly or whose answers are entailed by retrieved evidence; prompt adherence alone is imperfect.Source: Preventing Knowledge Leakage · p. 3
Constructing Thousands of Expert Domains
- Building thousands of curated domains manually is impractical, so the approach partitions an existing large dataset into clusters used as expert collections.The source dataset needs to support thousands of meaningful subcollections and provide queries with ground-truth answers or relevant documents.Source: Constructing Thousands of Expert Domains · pp. 3–4
Efficient Retrieval at Scale
- A single global index replaces thousands of separate indexes: global top-k results are assigned to clusters, which receive top-c documents for response generation.Clusters absent from the top-k receive a randomly sampled document because they are considered likely irrelevant to the query.Source: Efficient Retrieval at Scale · p. 4
- The global-index strategy preserves expert specialization while avoiding the overhead of thousands of separate search systems.It supports scalable simulation of expert LLMs for TREC-scale benchmarking.Source: Efficient Retrieval at Scale · p. 4
- The development set adds 342 Researchy Questions queries with GPT-o1 judge-generated three-level judgments; the test set has 50 NIST questions but only 37 ground-truth answers.The test answers were compared with model responses using GPT5-as-a-Judge to generate three-grained expertise labels.Source: Efficient Retrieval at Scale · p. 4
4 Implementation Details
The implementation builds a semantically tagged ClueWeb22 collection, clusters documents into domains, and simulates experts through shared-index retrieval. Query sets span short web searches and longer research questions, while expert labels are generated for development and test evaluation.
Document Collection
- The final semantically tagged ClueWeb22-derived collection contains 26,566,121 documents after combining tagged sources, removing duplicates, and filtering empty or missing content.The source intersections yielded 26,426,735 tagged MS MARCO-derived documents and 146,644 tagged research-question documents.Source: §4 · p. 4
Clustering
- Clustering starts with K-means over sparse Bing topic-tag vectors to create 1,500 clusters, then balances their sizes by splitting large clusters and merging small ones.The balancing procedure uses a minimum cluster size of 1,000 documents and preserves qualifying original clusters intact.Source: §4 · p. 5
Retrieval
- Instead of maintaining 1,131 cluster-specific indexes, the system searches one global index over 80.5 million documents using BM25 and selects up to three cluster documents from the global top 2,000.Clusters with no documents in those results receive a default template response.Source: §4 · p. 5
Query Collection
- The query collection combines 9,989 short MS MARCO web-search queries with 4,961 longer, exploratory Researchy Questions queries.MS MARCO queries were filtered to retain cases where adding a clicked document meaningfully changed the LLM answer; research questions were sampled from queries whose clicked documents appeared in the clusters.Source: §4 · pp. 5–6
Development Set
- The 342-query development set uses GPT-o1 comparisons with gold answers to assign three-level relevance judgments to model responses.These judgments reflect different degrees of expertise.Source: §4 · p. 6
Test Set
- NIST supplied 50 test questions, but ground-truth answers were available for 37, and only 16 of those questions elicited at least one relevant LM answer.GPT-5-as-a-Judge compared model responses with the ground-truth answers to generate three-graded expertise labels.Source: §4 · p. 6
- The evaluation prompt assigns scores 0, 1, and 2 for no knowledge, partial knowledge, and expertise, respectively.The judge compares each LM answer with the ground-truth answer for the question.Source: §4 · p. 7
5 Summary of Participating Systems
Participating groups explored classical expert retrieval, supervised and embedding-based routing, neural ranking, and modular systems. All submissions used the development set, but only six used it for training.
5 Summary of Participating Systems
- GenAIus found response-level evidence more effective than profile- or question-level representations, while SRCB combined LLM-generated pseudo-labels with efficient supervised ranking.GenAIus also tested filtering, query expansion, and reranking; SRCB compared ranking, classification, and regression formulations and reported strong performance from Qwen-based embedding models.Source: §5 · p. 7
- UDInfo’s hierarchical embedding-profile approach was strongest, and its results favored retrieval-style profiles over purely parametric classifiers.The approach combined cluster-centroid scoring with per-model nearest-neighbor refinement.Source: §5 · pp. 7–8
- IndeLab used modular neural systems and rank fusion, paulphys applied listwise learning-to-rank, and UvA IRLab trained LambdaRank using log-probability-derived soft labels.UvA IRLab’s ablations found global model-level statistics stronger predictors than query-specific semantic features in the current setting.Source: §5 · p. 8
Use of Metadata, External Models, and Development Data
- All 19 runs used the development set, but only six used it for training; four used an external LLM and three used the metadata package.The other 13 used the development set for validation, feature extraction, calibration, or tuning; 15 runs relied exclusively on provided resources.Source: Use of Metadata, External Models, and… · p. 8
6 Results
Submitted runs achieved nDCG@10 scores between 0.15 and 0.17, with the top systems predominantly using unsupervised techniques. Some development-set-trained runs were competitive, while others performed worse.
6 Results
- 0.147–0.152 nDCG@10: ensemble_v1, ensemble_v2, and ensemble_v3 were the most competitive of six development-set-trained submissions.Other development-set-trained runs performed worse; the authors suggest the 342-query devset and its distribution difference from test queries constrained generalization.Source: §6 · p. 8
- 0.17447 nDCG@10 for submission_4 led the runs, followed by infolab_UD_run2 at 0.16913 and r-ef-rm3-1000-ss at 0.16801.The strongest runs were predominantly unsupervised; the authors suggest response behavior signals may predict future LLM expertise.Source: §6 · p. 8, Table 1 · p. 9
7 Conclusion
The conclusion frames expert-model selection as a shift from document ranking toward selecting and orchestrating models. The track offers an initial controlled setting for studying behavioral inference of expertise.
7 Conclusion
- Information access is shifting from document ranking toward selecting and orchestrating expert models, a transition the authors say calls for scalable retrieval and comparison mechanisms.They anticipate ecosystems of millions of models with distinct expertise, behavior, and reliability.Source: §7 · p. 9
- The first track run created a controlled simulation of more than 1,000 domain-specialized LLMs, each constrained to its own document cluster.Results used 16 test queries and no human relevance judgments; the authors note dataset-establishment challenges.Source: §7 · p. 9