Source-linked AI summary
FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation
Shuai Wang, Ekaterina Khramtsova, Shengyao Zhuang, Guido Zuccon
TL;DR
Existing federated-search collections do not adequately represent modern RAG settings. FeB4RAG provides a RAG-oriented test collection, and the paper reports that naive RAG federated-search practices are far from optimal.
Problem
Existing collections poorly match RAG: API-dependent uncooperative engines risk obsolescence, adapted collections lack real-world engine diversity, and short keyword queries differ from richer natural-language requests.
Source: §2 · p. 4Method
FeB4RAG builds a RAG-oriented test collection through structured information-request generation and relevance labels aggregated from two open LLMs.
Source: §3.2 · p. 6, §3.3.2 · p. 8, §7 · p. 18Results
Current naive federated-search practices in RAG are far from optimal, motivating research into effective methods.
Source: §1 · p. 3Takeaways & Limitations
FeB4RAG provides a framework for exploring resource-selection strategies and result-merging techniques in RAG pipelines.
Source: §7 · p. 18Existing uncooperative collections depend on active search-engine APIs and risk obsolescence, while cooperative adaptations underrepresent real-world federation diversity.
Source: §2 · p. 4
Abstract
from arXiv · showhide
Federated search systems aggregate results from multiple search engines, selecting appropriate sources to enhance result quality and align with user intent. With the increasing uptake of Retrieval-Augmented Generation (RAG) pipelines, federated search can play a pivotal role in sourcing relevant information across heterogeneous data sources to generate informed responses. However, existing datasets, such as those developed in the past TREC FedWeb tracks, predate the RAG paradigm shift and lack representation of modern information retrieval challenges. To bridge this gap, we present FeB4RAG, a novel dataset specifically designed for federated search within RAG frameworks. This dataset, derived from 16 sub-collections of the widely used \beir benchmarking collection, includes 790 information requests (akin to conversational queries) tailored for chatbot applications, along with top results returned by each resource and associated LLM-derived relevance judgements. Additionally, to support the need for this collection, we demonstrate the impact on response generation of a high quality federated search system for RAG compared to a naive approach to federated search. We do so by comparing answers generated through the RAG pipeline through a qualitative side-by-side comparison. Our collection fosters and supports the development and evaluation of new federated search methods, especially in the context of RAG pipelines.
1 INTRODUCTION
Information scattered across resources motivates federated search, while RAG creates a renewed setting for combining retrieved information into answers. FeB4RAG is presented as a RAG-oriented collection for evaluating federated search, against the paper’s finding that naive RAG practices are far from optimal.
1 INTRODUCTION
- Information is distributed across organizational and personal resources, including websites, catalogs, databases, and cloud services.The passage describes this as information scattering.Source: §1 · p. 2
- In a RAG pipeline, a request triggers searches across resources, and an LLM synthesizes retrieved information into a response.The authors argue that growing RAG uptake, especially in chatbots, renews interest in federated search.Source: §1 · p. 2
- Federated search requires selecting resources for a request and merging their result lists, while naive round-robin merging can burden quality, cost, and latency.Searching irrelevant resources may increase hallucination risk and computational or network load.Source: §1 · p. 2
- FeB4RAG is tailored to RAG and uses LLM-generated relevance labels that the authors report have high agreement with human annotations.The paper also describes the collection’s construction and evaluation uses.Source: §1 · p. 2
- Current naive federated-search practices in RAG are far from optimal, motivating research into effective methods.FeB4RAG provides a means to evaluate those methods.Source: §1 · p. 3
2 LIMITATION OF AVAILABLE FEDERATED SEARCH COLLECTIONS
Existing federated-search collections have limitations in API stability, representation of real-world resource diversity, and fit with modern RAG requests and retrieval technology. These shortcomings motivate collections aligned with current technologies and user requirements.
2 LIMITATION OF AVAILABLE FEDERATED SEARCH COLLECTIONS
- API-dependent uncooperative collections risk obsolescence when search services change or disappear, undermining consistency and future adoption or expansion.The passage notes that eight engines were removed between TREC 2013 and 2014 due to service unavailability.Source: §2 · p. 4
- Cooperative collections adapted from a single dataset can evaluate shard selection but do not represent the diversity of specialized real-world search engines.The passage says this weakens their ability to mimic realistic federated scenarios.Source: §2 · p. 4
- Collections designed for ad-hoc web search diverge from RAG settings, whose requests are contextually richer and more specific than short keyword queries.RAG requests are more likely to resemble natural-language requests.Source: §2 · p. 4
- Older collections rely largely on early-2010s keyword matching or learning-to-rank, unlike the more sophisticated neural retrieval methods used by FeB4RAG.The passage cites contemporary neural information retrieval research as the contrast.Source: §2 · p. 4
- The authors argue that new collections should align with current technological advances and nuanced user requirements.They frame this alignment as important to keeping evaluation relevant to real-world applications.Source: §2 · p. 4
3 DATASET CREATION
FeB4RAG is built from BEIR collections and paired with retrieval models to represent federated search for RAG. Its creation process adapts datasets and queries to chatbot-style information requests, then labels retrieved results and search engines for evaluation.
3.1 Search Engine Selection
- The collection selects dense retrievers from MTEB for 13 datasets, excluding duplicate-question datasets and models barred by source-code, API, or computational constraints.TREC-News, Robust04, and Signal1M use the best-performing model among 30 dense retrievers evaluated for each dataset.Source: §3.1 · p. 5
- FeB4RAG uses 16 BEIR datasets spanning eight tasks and eight verticals, paired with nine retrieval models to simulate search engines.Model choices were based on MTEB effectiveness reported by 31 December 2023.Source: §3.1 · pp. 5–6
3.2 User Requests Creation
- Requests were manually selected when queries expressed clear information needs, while GPT-4 rewrote queries from datasets lacking explicit needs into conversational requests.Manual selection excluded ambiguous, confusing, overly short, or overly simplistic requests; rewritten requests were manually selected for varied expressions.Source: §3.2 · p. 6
- 790 user requests were included; Touché-2020 contributed 49 because its test set had 49 queries, and MS MARCO contributed 41 after manual exclusions.The target was roughly 50 requests per dataset.Source: §3.2 · p. 6
3.3 Relevance Labelling
- Original BEIR judgments were unsuitable because they could be sparse, use changed query contexts, omit results from other engines, or apply varied relevance scales.The collection labels individual results first, then aggregates those labels to determine search-engine relevance.Source: §3.3 · p. 8
3.3.1 Search Result Preparation.
- The relevance process retrieved each request’s top result from every selected engine, then assigned result labels using two open LLMs and aggregated their scores by averaging and rounding down.The four labels are key (3), high relevance (2), minimal relevance (1), and not relevant (0); the “nav” label was omitted.Source: §3.3.1 · p. 8, §3.3.2 · p. 8
3.3.3 Labelling Search Engines.
- Search-engine relevance is calculated from weighted result labels using graded precision, with scores ranging from 0 to 100.The weights are 0 for not relevant, 0.25 for minimal relevance, 0.5 for high relevance, and 1 for key.Source: §3.3.3 · p. 9
4 ANALYSIS OF RELEVANCE LABELLING
The analysis examines the collection’s relevance labels, their agreement with human and LLM judgments, and how relevant resources and verticals are distributed across requests. The results show substantial annotation agreement alongside variation in relevance across collections and a more balanced vertical distribution than in TREC FedWeb 2014.
4.1 Labelling Statistics
- Key and high relevance labels averaged 0.32% and 5.57%, respectively, consistent with stringent LLM labeling criteria.Non-relevant judgments made up 84.10% of results in NFCorpus, 81.82% in SciFact, and 80.16% in ArguAna; larger collections such as MS MARCO had a higher chance of relevance, though collection nature may also matter.Source: §4.1 · p. 9
4.2 LLM-based Labels vs. Human Annotations
- Across all datasets, Cohen’s Kappa exceeded 0.4; aggregated LLM labels reached 0.57, versus 0.54 for lgs-13b and 0.5 for solar-11b.The comparison used a sample of human-judged results and binarized LLM labels to align with the heterogeneous BEIR scales.Source: §4.2 · pp. 11–12
4.3 Agreements between LLMs
- The two LLMs achieved an overall Cohen’s Kappa of 0.57 across 7,901,610 comparisons, near the 0.6 reported for two TREC FedWeb 2013 human annotators [46].The authors describe this as moderate agreement.Source: §4.3 · p. 12
4.4 Importance of each Resource
- On average, 11.9 search engines contained relevant content per request, while only a handful of requests had a single relevant resource.Figure 3 reports the number of queries for which n search engines contain relevant information.Source: §4.4 · p. 12, Figure 3 · p. 12
4.5 Importance of Resource Vertical
- FeB4RAG’s best-resource graded-precision distribution is more balanced across verticals than TREC FedWeb 2014’s, mitigating the risk of vertical-based selection bias going undetected.The comparison categorizes each request by the vertical of its highest-graded-precision resource.Source: §4.5 · p. 13, Figure 4 · p. 13
5 USE OF THE COLLECTION
FeB4RAG supports evaluation of federated search tasks in RAG, including resource selection and result merging, and is designed to expand as the collection and RAG research develop.
5 USE OF THE COLLECTION
- FeB4RAG is intended to evaluate resource selection and result merging and to support future tasks arising from RAG integration.The paper identifies these as practical uses of the collection.Source: §5 · p. 13
5.1 Resource Selection
- The collection supports evaluation of resource selection by ranking or classifying its 16 search engines.The paper also describes uncooperative methods that rely on samples and cooperative methods that use collection statistics.Source: §5.1 · p. 14
5.2 Result Merging
- Result merging effectiveness depends on resource selection; nDCG@k can be used, but RAG-suited evaluation measures remain an open research direction.The collection can support alternative evaluation paradigms.Source: §5.2 · p. 14
5.3 Expanding the Collection
- An expandable codebase supports adding requests, retrievers, relevance assessors, and full-RAG evaluation.The current choice of dense retrievers over more computationally expensive search technologies, such as zero-shot LLM rankers, reflects computational budget constraints.Source: §5.3 · p. 15
6 DEMONSTRATING THE NEED FOR FEB4RAG
The paper compares RAG responses using a naive federation strategy and a simulated best-fed strategy, evaluating human preferences across criteria and BEIR datasets. Best-fed responses are preferred more often overall, though the pattern varies by criterion and dataset.
6 DEMONSTRATING THE NEED FOR FEB4RAG
- The experiment demonstrates the value of a tailored federated search approach in a RAG pipeline compared with naive resource federation.The paper frames this comparison as evidence for the need for FeB4RAG.Source: §6 · p. 15
6.1 RAG Systems with Federated Search
- Naive-fed searches every engine and round-robin merges results, while best-fed selects engines with positive graded precision and merges minimally relevant results by relevance.The resulting top-k list is supplied to an LLM to generate a response.Source: §6.1 · pp. 15–16
6.2 Evaluation Methodology
- Human assessors compare responses side by side using coverage, consistency, correctness, and clarity, with randomized presentation order to mitigate bias.Assessors could choose no preference when neither response was clearly better.Source: §6.2 · pp. 16–17
- The human comparison samples 80 requests—five from each BEIR dataset—although generated responses for the full request set are included in the collection.The comparison uses responses generated by the two simulated systems.Source: §6.2 · p. 17
6.3 Result Analysis
- Best-fed responses are preferred more often than naive-fed responses overall and across criteria, with marginal differences for clarity and consistency.Coverage shows the greatest preference difference, followed by a smaller but comparatively sizeable difference for correctness.Source: Figure 7 · p. 18, §6.3 · p. 18
- Best-fed receives more preferences than naive-fed on most datasets, but Climate-FEVER is an exception; best-fed preferences occur across all datasets.Naive-fed preferences do not occur across all datasets, including TREC-COVID and ArguAna.Source: §6.3 · p. 18
7 CONCLUSION
FeB4RAG is a test collection for evaluating federated search in RAG pipelines, built around information requests and resources drawn from BEIR.
7 CONCLUSION
- FeB4RAG provides a framework for exploring resource selection and result merging within RAG pipelines.Source: §7 · p. 18
- The collection contains 790 information requests intended to better represent what users submit to RAG applications than requests in existing collections.Source: §7 · p. 19
- Its 16 BEIR-based search engines use state-of-the-art dense retrieval methods, with LLM relevance judgments at search-result and search-engine levels and an expandable codebase.The codebase can be adapted to integrate more assessments, engines, and requests.Source: §7 · p. 20