PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search

Shuai Wang, Shengyao Zhuang, Bevan Koopman, Guido Zuccon

arXiv:2401.17645v1cs.IRcs.AI

TL;DR

Resource selection in federated search can improve relevance while reducing query costs, but existing approaches often require labor-intensive human labels. ReSLLM uses LLM prompting for zero-shot resource selection and SLAT fine-tunes it with synthetic labels from logged queries and snippets. The paper reports competitive zero-shot results and SLAT-tuned effectiveness comparable to supervised baselines, while noting that downstream RAG impact was not directly evaluated.

  • Problem

    Resource selection methods often rely on human-annotated relevance labels, whose creation is costly and time-consuming, despite selection’s role in federated-search quality and query costs.

    Source: §1 · p. 2

  • Method

    ReSLLM prompts an LLM to select resources, while SLAT uses LLM-generated relevance assessments for logged query-snippet pairs to create resource-level training labels and fine-tune ReSLLM.

    Source: §1 · p. 2

  • Results

    ReSLLM outperforms several baselines and matches supervised-model efficacy; configured zero-shot backbones beat unsupervised and embedding-based baselines, and SLAT-tuned ReSLLM is comparable to supervised baselines.

    Source: §1 · p. 2, §6 · p. 16

  • Takeaways & Limitations

    ReSLLM and SLAT provide resource-selection approaches that do not require human-generated training labels.

    Source: §1 · p. 2, §6 · p. 16

    The study did not directly evaluate effects on downstream RAG generation because evaluation resources for that setting were unavailable.

    Source: §6 · p. 16

Abstract

from arXiv · show

Federated search, which involves integrating results from multiple independent search engines, will become increasingly pivotal in the context of Retrieval-Augmented Generation pipelines empowering LLM-based applications such as chatbots. These systems often distribute queries among various search engines, ranging from specialized (e.g., PubMed) to general (e.g., Google), based on the nature of user utterances. A critical aspect of federated search is resource selection - the selection of appropriate resources prior to issuing the query to ensure high-quality and rapid responses, and contain costs associated with calling the external search engines. However, current SOTA resource selection methodologies primarily rely on feature-based learning approaches. These methods often involve the labour intensive and expensive creation of training labels for each resource. In contrast, LLMs have exhibited strong effectiveness as zero-shot methods across NLP and IR tasks. We hypothesise that in the context of federated search LLMs can assess the relevance of resources without the need for extensive predefined labels or features. In this paper, we propose ReSLLM. Our ReSLLM method exploits LLMs to drive the selection of resources in federated search in a zero-shot setting. In addition, we devise an unsupervised fine tuning protocol, the Synthetic Label Augmentation Tuning (SLAT), where the relevance of previously logged queries and snippets from resources is predicted using an off-the-shelf LLM and then in turn used to fine-tune ReSLLM with respect to resource selection. Our empirical evaluation and analysis details the factors influencing the effectiveness of LLMs in this context. The results showcase the merits of ReSLLM for resource selection: not only competitive effectiveness in the zero-shot setting, but also obtaining large when fine-tuned using SLAT-protocol.

1 INTRODUCTION

Federated search routes queries across independent resources, so selecting relevant engines can improve result quality while reducing query time and cost. ReSLLM uses LLM prompting without human labels, and SLAT fine-tunes it using synthetic relevance labels; experiments report competitive zero-shot effectiveness and results comparable to supervised baselines after tuning.

1 INTRODUCTION

  • Federated search integrates results from multiple search engines or databases, including sources with distinct specializations such as Amazon and arXiv.The unified view is useful when information is dispersed across sources.Source: §1 · p. 1
  • Effective resource selection targets engines likely to contain relevant documents, improving retrieved information while reducing API-query time and costs.The motivation also applies to RAG pipelines, where only a handful of retrieved results may be handled by the LLM.Source: §1 · p. 2
  • ReSLLM prompts an LLM to select resources using their names or descriptions, without human-labeled training data.SLAT uses an off-the-shelf LLM to label logged query-result snippets, aggregates those labels by resource, and fine-tunes ReSLLM.Source: §1 · p. 2
  • On TREC FedWeb 13 and 14, configured zero-shot ReSLLM backbones outperform unsupervised and embedding-based baselines, while SLAT-tuned ReSLLM is comparable to supervised baselines.The supervised baselines typically rely on human-generated labels.Source: §1 · p. 2
  • Backbone size and architecture matter more for zero-shot ReSLLM than for ReSLLM fine-tuned with SLAT.The introduction reports that their importance diminishes after SLAT fine-tuning.Source: §1 · p. 3

2 RELATED WORK

Prior resource selection methods include unsupervised approaches based on resource statistics or sampled documents and supervised approaches that learn from features or relevance labels. LLMs offer zero-shot ranking techniques that this work adapts to rank resources.

2.1 Resource Selection

  • Unsupervised resource selection includes lexicon-based methods and sample-based methods such as ReDDE, CRCS, and SUSHI, while KBCS enriches entity-based query-resource similarity.Embedding-based methods also use semantic analysis, though earlier unsupervised methods may not capture textual structure fully.Source: §2.1 · p. 3
  • Supervised resource selection spans classification and learning-to-rank, but methods such as SVMrank and LTRRS rely on feature engineering.The cited approaches use features including term statistics and centralized-sample-index metrics.Source: §2.1 · p. 3

3 METHODOLOGY

ReSLLM prompts an instruction-fine-tuned LLM to judge resources for a query, then ranks them using its yes/no response probabilities. SLAT extends the approach by generating synthetic relevance labels from logged query-snippet pairs and using aggregated resource labels to fine-tune ReSLLM.

3 METHODOLOGY

  • ReSLLM combines LLM-based resource selection with optional SLAT fine-tuning and alternative resource representations.Figure 1 overviews the method and tuning protocol.Source: §3 · p. 4, Figure 1 · p. 5

3.1 Problem Formulation

  • The selector maps a query and a set of resources to a suitable subset containing engines likely to return relevant documents.The paper treats resources as search engines, while noting the formulation could extend to databases and knowledge bases.Source: §3.1 · p. 4
  • Resource selection can be constrained to return a ranked top-k subset, and the setting assumes uncooperative resources expose descriptions and results rather than collection statistics.A log of earlier queries and returned snippets may be available.Source: §3.1 · p. 4

3.2 The ReSLLM Prompting Method

  • ReSLLM asks an instruction-fine-tuned LLM whether a resource represented in the prompt is appropriate for the user query.The prompt includes system instructions, resource representation, user input, and task instructions; representation options include a name, URL, or description, with query and sampled snippets as inputs.Source: §3.2 · p. 4, Table 1 · p. 6
  • ReSLLM converts the LLM’s yes/no response logits into probabilities and scores each resource by the difference between its yes and no probabilities.Resources are ranked in descending score order; the paper notes that practice may restrict selection to nonnegative scores when fewer than k resources qualify.Source: §3.2 · p. 5

3.3 The SLAT Protocol

  • SLAT uses logged queries and resource snippets to create synthetic relevance assessments, aggregate them into resource-level scores, and fine-tune the selector.The protocol is motivated by settings where human relevance labels are unavailable or inconvenient to obtain.Source: §3.3 · pp. 5–6

3.3.1 Automatic Relevance Assessment.

  • SLAT adapts an LLM relevance-assessment prompt to assign five levels matching TREC FedWeb: Navigational, Key, Highly Relevant, Relevant, and Non-Relevant.The prompt is adapted from Thomas et al. [37] and uses an open-source backbone rather than GPT-4.0.Source: §3.3.1 · pp. 6–7, Table 2 · p. 7

3.3.2 Aggregation of Relevance to the Resource Level.

  • SLAT aggregates query-snippet assessments using graded precision at the top 10, with TREC FedWeb 2013 weights, then scales and rounds scores to integers from 0 to 100.The chosen weights align with the evaluation collection and can be adapted to other contexts.Source: §3.3.2 · pp. 7–8

4.1 Considered LLMs

The experiments use seven LLMs spanning encoder-decoder and decoder-only architectures and multiple model sizes, with Flan-T5-XL as the default backbone.

4.1 Considered LLMs

  • The model set includes Flan-T5-Large, XL, and XXL plus Llama2-7B-Chat and 13B-Chat, alongside two open-source models selected under a 13-billion-parameter compute limit.The additional models were chosen by average perplexity-reduction effectiveness on the leaderboard; architecture and size were not selection criteria.Source: §4.1 · pp. 8–9

4.2 Datasets

The experiments use FedWeb-13 and FedWeb-14, collections of search engines with test queries, retrieved snippets, and relevance assessments. A separate query log supports SLAT and an embedding baseline, while descriptions associated with FedWeb queries provide conversational-query inputs.

4.2 Datasets

  • FedWeb-13 covers 157 search engines and FedWeb-14 covers 149, with test queries, top-10 snippets, and relevance assessments in both collections.FedWeb-13 has 50 test queries; FedWeb-13 and -14 include 2,000 and 4,000 single-term synthetic queries, respectively.Source: §4.2 · p. 9
  • The query log combines 406 additional queries and snippets with unlabelled single-term queries and snippets from FedWeb-13/2014.It is used to investigate SLAT fine-tuning and to create the unsupervised embedding baseline.Source: §4.2 · p. 9
  • Conversational queries are represented by the descriptions associated with FedWeb queries, while ad-hoc queries use the collection queries themselves.Source: §4.2 · p. 9

4.3 Evaluation measures

The evaluation uses normalized ranking metrics at multiple cut-offs, including nP to assess the relevance of top-ranked resources.

4.3 Evaluation measures

  • nDCG is evaluated at cut-offs 10, 20, and 100, while normalized Graded Precision (nP) is evaluated at cut-offs 1 and 5.Source: §4.3 · p. 9
  • nP@k normalizes Graded Precision at k by the highest Graded Precision achievable by any resource for that topic at the same cut-off.Source: §4.3 · p. 9

4.4 Baselines

The comparison includes supervised and unsupervised TREC FedWeb runs, alongside an embedding baseline constructed from logged snippets. Some selected TREC baselines use data unavailable to ReSLLM, including test-set annotations for one supervised run.

4.4 Baselines

  • The baselines were selected from TREC FedWeb submissions, with supervised runs relying on relevance assessments used to train a learning model.The supervised baseline was selected using the best nDCG@20 leaderboard performance.Source: §4.4 · p. 9
  • The supervised baseline is ECNUCS-ecomsvz from TREC FedWeb 2014, selected for the best nDCG@20 among supervised runs.It uses human-annotated labels and incorporates Google search and fused approaches; the paper notes it also used test-set annotations to select resources.Source: §4.4 · pp. 9–10
  • The unsupervised baselines are UPD-UPDFW13mu for FedWeb 2013 and Dragon-drexelRS7 for FedWeb 2014.They implement TWF-IRF and SUSHI, respectively; Dragon-drexelRS7 uses additional query-log data.Source: §4.4 · p. 10
  • ECNUCS-ecomsvz and Dragon-drexelRS7 use additional data unavailable to ReSLLM, including commercial market share, external Google API access, or sampled documents.Source: §4.4 · p. 10
  • The embedding baseline encodes queries and logged snippets with BAAI/bge-large-en-v1.5 [45], scoring each resource by the average similarity of its top three snippets.Resource scores use cosine similarity between query and snippet encodings.Source: §4.4 · p. 10

4.5 SLAT Settings

SLAT uses an LLM to create synthetic relevance assessments from logged query-snippet pairs, then uses them to fine-tune ReSLLM. For conversational-query fine-tuning, the authors generate descriptions from logged queries and relevant snippets because most logged queries are single terms.

4.5.1 LLM used in SLAT for synthetic relevance assessments generation.

  • SOLAR-10.7B-Instruct-v1.0 generated SLAT relevance assessments after GPT-4.0 was deemed too costly and slow and several open models produced low-quality assessments.Agreement between SOLAR assessments and each of two TREC assessors was κ=0.35; agreement between the assessors was κ=0.6.Source: §4.5.1 · p. 10

4.5.2 Training queries for SLAT & generation of conversational queries.

  • Conversational-query fine-tuning does not reliably use logged queries directly because most are single-term queries, unlike the long descriptions used as conversational queries.The authors instead use SOLAR to judge logged snippets, sample three navigationally relevant snippets per query, and generate a query description from the ad-hoc query and snippets.Source: §4.5.2 · pp. 10–11
  • The conversational-query prompt asks for a description of the information need using an ad-hoc query and three navigationally relevant snippets.The example pair was manually constructed to avoid using test data.Source: Table 3 · p. 11, §4.5.2 · p. 11

4.5.3 Fine-tuning Settings for SLAT.

  • Resource labels are assigned from aggregated scores using thresholds of ≥50 for highly relevant, 25 to 50 for marginally relevant, and <25 for not relevant.Source: §4.5.3 · p. 11
  • SLAT fine-tunes Flan-large, Flan-xl, and LlaMa2-7b-chat, with two epochs for the Flan models and one for LlaMa2-7b-chat.The Flan models use contrastive loss; batch sizes are 20 for Flan-large, 5 for Flan-xl, and 2 with gradient accumulation of 5 for LlaMa2-7b-chat.Source: §4.5.3 · p. 11

5 RESULTS AND ANALYSIS

Zero-shot ReSLLM’s effectiveness depends on its LLM backbone, while SLAT fine-tuning substantially improves results and reduces observed differences between model sizes and architectures. Resource representation also matters differently before and after tuning.

5.1 Zero-shot ReSLLM

  • Solar-11b generally outperforms other LLMs, and Solar-11b and Mixtral-13b improve over Embedding across most metrics, while other tested backbones trail supervised baselines.Solar-11b’s exceptions are nDCG@20 and nP@5 on FedWeb-13 ad-hoc queries, where Mixtral-13b is marginally higher; ReSLLM with Solar-11b exceeds unsupervised baselines.Source: §5.1 · pp. 12–13
  • Zero-shot ReSLLM performs better on conversational than ad-hoc queries, whereas SLAT+ReSLLM shows no significant conversational-query improvement over ad-hoc queries.The authors note that converting logged ad-hoc queries into conversational queries may alter intent and misalign generated labels with conversational context.Source: §5.1 · p. 12, §5.2 · p. 14

5.2 ReSLLM with SLAT

  • SLAT+ReSLLM achieves the highest effectiveness among baselines except on some FedWeb-14 metrics versus the supervised baseline, and reaches effectiveness on par with that baseline without human annotations.Table 5 compares SLAT+ReSLLM with zero-shot ReSLLM and baselines; SLAT uses LLM-generated labels for fine-tuning.Source: §5.2 · pp. 13–15, Table 5 · p. 13

5.3 Impact of LLM Size and Architecture

  • 32% and a further 60%: zero-shot nDCG@20 increases on average from flan-large to flan-xl, then flan-xxl; after SLAT, flan-large and flan-xl differ by only 1% on average.SLAT fine-tuning results for flan-xxl were unavailable because of computational-resource limits.Source: Figure 2 · p. 13, §5.3 · p. 15
  • In zero-shot selection, flan-xl significantly outperforms decoder-only LlaMa2-7b-chat, particularly on nDCG@20; the gap narrows markedly after SLAT fine-tuning.The comparison concerns encoder-decoder flan-xl and decoder-only LlaMa2-7b-chat.Source: Figure 3 · p. 14, §5.3 · p. 15

5.4 Impact of Resource Representation

  • In zero-shot ReSLLM, N+D improves effectiveness by approximately 6%, N+S by 3.5%, and N+S+D by 1.1%; after SLAT, N+S decreases effectiveness by approximately 35% and N+S+D by 16%.After SLAT, N+D differs by less than 1%; the paper describes richer representations as less advantageous after tuning.Source: Figure 4 · p. 14, §5.4 · pp. 15–16

6 CONCLUSION

The paper introduces ReSLLM, a zero-shot resource selector, and SLAT, which fine-tunes it with LLM-generated labels without human intervention. Experiments show effective resource selection, while direct impact on RAG generation remains untested.

6 CONCLUSION

  • ReSLLM and SLAT+ReSLLM outperform several baselines and match supervised models in certain scenarios.SLAT uses LLM-generated synthetic labels to fine-tune ReSLLM without human intervention.Source: §6 · p. 16
  • ReSLLM selects federated-search resources without human-labeled data, and SLAT fine-tunes it using LLM-generated synthetic labels.Source: §6 · p. 16
  • The study did not directly evaluate downstream RAG impact because suitable evaluation resources were unavailable; creating federated-search datasets is a proposed next step.Source: §6 · p. 16
Loading 2401.17645v1…