PaperlayerView exact revision on arXiv

Source-linked AI summary

Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers

Kunal Sawarkar, Abhilasha Mangal, Shivam Raj Solanki

arXiv:2404.07220v2cs.IRcs.AIcs.CL

TL;DR

As RAG corpora scale, limited retrieval methods can feed irrelevant context and reduce answer accuracy. Blended RAG combines semantic-search indexes with hybrid queries, improving retrieval across benchmarks and achieving 68% F1 on SQUAD and 42% EM on NQ.

  • Problem

    Existing RAG pipelines often rely on keyword or similarity-based retrieval, while finding the best search method remains an open research area.

  • Method

    Blended RAG combines BM25, dense-vector, and sparse-encoder indexes with hybrid query strategies to select effective retrievers for RAG.

  • Results

    Across benchmark tasks, Blended RAG improves retrieval and generative QA, reaching 68% F1 Score on SQUAD and 42% EM Score on NQ.

  • Takeaways & Limitations

    Blended RAG generalizes across multiple datasets without task-specific training or few-shot exemplars, supporting zeroshot generative QA.

  • Takeaways & Limitations

    Finding the best search method for RAG remains an emerging area of research.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) is a prevalent approach to infuse a private knowledge base of documents with Large Language Models (LLM) to build Generative Q\&A (Question-Answering) systems. However, RAG accuracy becomes increasingly challenging as the corpus of documents scales up, with Retrievers playing an outsized role in the overall RAG accuracy by extracting the most relevant document from the corpus to provide context to the LLM. In this paper, we propose the 'Blended RAG' method of leveraging semantic search techniques, such as Dense Vector indexes and Sparse Encoder indexes, blended with hybrid query strategies. Our study achieves better retrieval results and sets new benchmarks for IR (Information Retrieval) datasets like NQ and TREC-COVID datasets. We further extend such a 'Blended Retriever' to the RAG system to demonstrate far superior results on Generative Q\&A datasets like SQUAD, even surpassing fine-tuning performance.

I. INTRODUCTION · II. RELATED WORK · A. Limitations in the current RAG system

RAG combines dynamically retrieved external knowledge with generative language models, making retriever quality fundamental to answer accuracy. The paper addresses limitations of keyword- and similarity-based retrieval by pursuing semantic-search retrievers and hybrid queries.

  • I. INTRODUCTION: RAG combines patterns learned during training with dynamically retrieved external knowledge to support text generation.Its effectiveness depends fundamentally on the Retriever (R) and Generator (G), including the LLM’s size and type.
  • I. INTRODUCTION: The Retriever sifts through documents for relevant information that grounds and enriches the Generator’s answers.The passage characterizes the retriever as the research component that supplies contextually grounded text.
  • II. RELATED WORK: BM25 ranks documents using Term Frequency, Inverse Document Frequency, and Document Length to calculate query relevance.BM25 is presented as a historically important similarity-search method in information retrieval.
  • II. RELATED WORK: Dense vector models use KNN and vector proximity measures such as cosine similarity to capture deep semantic relationships in large-scale search.The cited work describes assessing kinship between queries and data entities through vectorial proximity.
  • II. RELATED WORK: Sparse encoder vector models represent document semantics precisely while efficiently handling high-dimensional data and preserving interpretability.Sparse Encoder indexes map documents and user queries into extensive arrays of associated corpus-derived terms.
  • A. Limitations in the current RAG system: Most current RAG retrieval methods rely on keyword and similarity-based searches, which can restrict overall system accuracy.The paper identifies these methods as the current retrieval baseline and introduces Table 1 as a benchmark summary.
  • A. Limitations in the current RAG system: Tweaking Generator prompts or tuning has limited impact when the Retriever supplies irrelevant context, because the resulting answer becomes inaccurate.The paper therefore frames retriever improvement as necessary rather than focusing only on the Generator.
  • A. Limitations in the current RAG system: The study aims to improve retriever and RAG accuracy by incorporating Semantic Search-Based Retrievers and Hybrid Search Queries.It presents finding the best search method for RAG as an emerging research area.

III. BLENDED RETRIEVERS · A. Methodology

The study develops Blended Retrievers by combining keyword, dense-vector, and semantic sparse-encoder search with hybrid queries. Its methodology progresses from BM25 matching to hybrid retrieval across multiple indices and evaluates query types using top-k retrieval accuracy.

  • III. BLENDED RETRIEVERS: The approach combines keyword-based similarity, dense-vector search, and semantic sparse encoders into hybrid queries for RAG retrieval.The primary indices are BM25, KNN, and Elastic Learned Sparse EncodeR.
  • III. BLENDED RETRIEVERS: BM25 supports full-text search with fuzzy matching, while the dense vector index uses sentence transformers to compare document and query representations.The dense index identifies proximity between vector representations derived from document and query content.
  • A. Methodology: The methodology begins with an elementary BM25 match query and then escalates to hybrid queries spanning diverse search techniques and multiple fields.This progression addresses cases where the query text’s exact document location is unknown.
  • A. Methodology: Multi-match queries include Cross Fields for concurrence, Most Fields for varied field representations, and Best Fields for aggregating words within one field.These categories organize how text is matched across or within fields.
  • A. Methodology: Phrase Prefix resembles Best Fields but prioritizes phrases over keywords, after which dense-vector KNN and sparse-encoder indices are added with bespoke hybrid queries.The strategy synthesizes the strengths of each index to refine retrieval accuracy.
  • A. Methodology: The study calculates top-k retrieval accuracy to distill the performance of each query type.This metric is used after incorporating the BM25, dense-vector, and sparse-encoder retrieval strategies.
  • A. Methodology: Figure 1 presents a scheme for creating Blended Retrievers by blending semantic search with hybrid queries.The scheme represents the proposed integration of semantic search and hybrid-query strategies.

B. Constructing RAG System · IV. EXPERIMENTATION FOR RETRIEVER EVALUATION

The study constructs and evaluates RAG retrievers by testing diverse hybrid query formulations across four benchmark datasets. Top-10 retrieval accuracy identifies six blended retrievers for further comparison.

  • B. Constructing RAG System: The evaluated hybrid queries combined different index types with query strategies.The selected queries represented a synthesis of query strategies aligned with various index types.
  • B. Constructing RAG System: A select sextet of hybrid queries exhibiting the strongest retrieval efficacy was chosen for further scrutiny.These six queries were described as the culmination of retriever experimentation.
  • IV. EXPERIMENTATION FOR RETRIEVER EVALUATION: The six selected blended retrievers were compared for retriever evaluation across benchmark datasets.The comparison focused on the precision of the retrieval component within RAG.
  • B. Constructing RAG System: Hybrid query formulations were evaluated across NQ, TREC-COVID, SqUAD, and HotPotQA.The evaluation examined retriever performance across benchmark datasets with diverse and complex source landscapes.
  • B. Constructing RAG System: The RAG construction process involved scrutinizing a myriad of hybrid query formulations.This comprehensive evaluation addressed the multi-fold challenge of constructing an effective RAG system.
  • IV. EXPERIMENTATION FOR RETRIEVER EVALUATION: Top-10 retrieval accuracy narrowed the candidates to six blended retriever types for each benchmark dataset.Each candidate consisted of an index and a hybrid query.

1) Top-10 retrieval accuracy on the NQ dataset : · 2) Top-10 Retrieval Accuracy on TREC-Covid dataset:

On NQ, Sparse Encoder hybrid queries using Best Fields achieved the highest top-10 retrieval accuracy at 88.77%. On TREC-COVID, vector-search hybrid queries outperformed keyword-based approaches, reaching 78% for relevance score 1 and 98% for score 2.

  • 1) Top-10 retrieval accuracy on the NQ dataset :: 88.77% top-10 retrieval accuracy was achieved on NQ by the Sparse Encoder hybrid query using Best Fields.This result surpassed all other formulations and established a new retrieval benchmark for the dataset.
  • 1) Top-10 retrieval accuracy on the NQ dataset :: Hybrid query strategies improved NQ retrieval by utilizing multiple data fields effectively.The analysis identified hybrid queries with Sparse Encoder and Best Fields as the strongest formulation.
  • 2) Top-10 Retrieval Accuracy on TREC-Covid dataset:: TREC-COVID relevancy scores ranged from -1 to 2, with -1 indicating irrelevance and 2 indicating high relevance.Initial assessments focused on documents assigned relevancy score 1, which was considered partially relevant.
  • 2) Top-10 Retrieval Accuracy on TREC-Covid dataset:: TREC-COVID score-2 evaluation tested the same query spectrum on documents fully pertinent to their associated queries.The score-2 setting represented documents that fully met the relevance criteria.
  • 2) Top-10 Retrieval Accuracy on TREC-Covid dataset:: 98% top-10 retrieval accuracy was achieved on TREC-COVID score 2 by the Sparse Encoder hybrid query using Best Fields.This result eclipsed all other formulations and reinforced the efficacy of vector-search hybrid queries over conventional keyword-based methods.

3) Top-10 Retrieval Accuracy on the HotPotQA dataset :

Evaluation on HotPotQA was limited to a select subset of hybrid queries because the dataset contains over 5M documents and 7,500 queries, but the results still showed strong performance. Sparse EncodeR blended with Best Fields achieved the highest efficiency at 65.70%.

  • Evaluation constraints: Evaluation was confined to a select subset of hybrid queries because HotPotQA contains over 5M documents and 7,500 queries, creating substantial compute requirements.The analysis was therefore limited, with results presented in Figure 5.
  • Hybrid-query results: 65.70% efficiency was achieved by the hybrid query blending Sparse EncodeR with Best Fields, the highest-performing approach on HotPotQA.Cross Fields and Best Fields hybrid strategies demonstrated superior performance overall.

A. Retriever Benchmarking … 3) SqUAD Dataset Benchmarking:

The study benchmarks sextet-query retrievers on NQ and TREC-COVID using NDCG@10, then evaluates retrieval accuracy on SQuAD. Blended and semantic-search-based approaches outperform the reported benchmarks and traditional or sparse retrieval methods in the supplied results.

  • A. Retriever Benchmarking: Sextet queries were selected from the best Index + Query combinations for IR benchmarking with NDCG@10.NDCG@10 is the Normalised Discounted Cumulative Gain metric.
  • 1) NQ dataset benchmarking:: 0.67 NDCG@10 was achieved on NQ, 5.8% above monoT5-3B’s 0.633 benchmark.All semantic search-based hybrid queries also outperformed the current benchmark.
  • 1) NQ dataset benchmarking:: Semantic search-based hybrid queries were identified as stronger candidates for developing the RAG pipeline because they exceeded the NQ benchmark.The reported benchmark was achieved by monoT5-3B.
  • 2) TREC-Covid Dataset Benchmarking :: 0.87 NDCG@10 was achieved on TREC-COVID, an 8.2% increase over the COCO-DR Large benchmark of 0.804.The passage describes the hybrid-query suite as exceeding the current 0.80 NDCG@10 benchmark.
  • 3) SqUAD Dataset Benchmarking:: SQuAD retrieval accuracy was evaluated by constructing a title-and-context corpus and indexing it with BM25, dense vector, and Sparse Encoder methods.SQuAD was evaluated for consistency even though it is not an IR dataset.
  • 3) SqUAD Dataset Benchmarking:: Dense vector (KNN)-based semantic searches achieved higher SQuAD retrieval accuracy than sparse vector-based and traditional similarity-based searches for k=5, 10, and 20.The comparison concerns top-k retrieval performance.

B. Summary of Retriever Evaluation · V. RAG EXPERIMENTATION

Blended Retrieval improves retrieval accuracy across NQ, TREC-COVID, and SQUAD, though not CoQA, with hybrid queries and semantic search driving the gains. The paper then extends the best index–query retriever to RAG experiments using FLAN-T5-XXL.

  • B. Summary of Retriever Evaluation: Retrieval accuracy improves across all evaluated datasets except CoQA, attributed to hybrid queries’ use of metadata to find pertinent results.Evaluation used Top-k metrics for k ∈ {5, 10, 20} across NQ, TREC-COVID, SQUAD, and CoQA.
  • B. Summary of Retriever Evaluation: Dense vector-based KNN semantic search markedly outperforms keyword-based search approaches.
  • B. Summary of Retriever Evaluation: Semantic-search hybrid queries achieve better retrieval precision than conventional keyword-based or vector-based searches.
  • B. Summary of Retriever Evaluation: Sparse Encoder semantic search combined with the Best Fields hybrid query often produces superior results to other evaluated methods.
  • V. RAG EXPERIMENTATION: The evaluation identifies the best retriever as the best combination of indices and query for extension into the RAG pipeline.
  • V. RAG EXPERIMENTATION: All RAG experiments use FLAN-T5-XXL to avoid effects from differences in LLM size or type.

A. RAG Evaluation on the SqUAD Dataset · B. RAG Evaluation on the NQ Dataset

The study evaluates Blended RAG on SqUAD and NQ using answer-generation and retrieval metrics. Blended RAG improves reported performance without dataset-specific fine-tuning, including a 50% higher F1 score on SqUAD and an EM of 42.63 on NQ.

  • A. RAG Evaluation on the SqUAD Dataset: SqUAD evaluation compares RAG-original, RAG-end2end, and Blended RAG using Exact Match, F1, Top-5, and Top-10 metrics.RAG-original was fine-tuned on Natural Questions, while RAG-end2end received additional SQuAD domain-adaptation fine-tuning.
  • A. RAG Evaluation on the SqUAD Dataset: Blended RAG was not trained on SQuAD or related corpora and instead uses optimized field selections, hybrid query formulations, and semantic indices.The retrieved context is supplied to LLMs for answer generation.
  • A. RAG Evaluation on the SqUAD Dataset: F1 scores higher by 50%: Blended RAG improves Generative Q&A performance on SqUAD without dataset-specific fine-tuning.The paper identifies this as advantageous for large enterprise datasets where fine-tuning may be impractical or unfeasible.
  • B. RAG Evaluation on the NQ Dataset: NQ evaluation tests Blended RAG with zero-shot learning against other nonfine-tuned models using Exact Match, F1 Score, Top-5, and Top-20.The metrics assess answer generation and retrieval accuracy.
  • B. RAG Evaluation on the NQ Dataset: EM of 42.63: Blended RAG (Zero-shot) demonstrates superior performance on NQ.The passage reports this result as improving the prior benchmark by 35%.
  • B. RAG Evaluation on the NQ Dataset: Improving the prior benchmark by 35%: Blended RAG (Zero-shot) establishes the reported NQ comparison result.This improvement accompanies the reported EM of 42.63.

VI. DISCUSSION · A. Trade-off between Sparse and Dense Vector Indices · B. Blended Retrievers without Metadata

The discussion connects Retriever and RAG evaluation while identifying operational trade-offs between dense and sparse indexes. It also finds that metadata affects blended-retriever effectiveness, with sparse semantic search remaining strongest on metadata-poor CoQA and current metrics poorly reflecting human alignment.

  • VI. DISCUSSION: The study seeks synergy between Retriever evaluation in information retrieval and RAG evaluation in the LLM domain, despite lacking standard datasets benchmarking both.The authors identify this separation and dataset gap as ongoing challenges.
  • A. Trade-off between Sparse and Dense Vector Indices: 50GB dense indexing for HotPotQA’s 5M documents hampers processing efficiency, illustrating the computational burden of large-scale dense retrieval.The corpus generates a dense vector index of approximately 50GB.
  • A. Trade-off between Sparse and Dense Vector Indices: Dense indexes provide faster indexing but slower querying, whereas sparse indexes index more slowly while querying faster.The discussion frames these opposing indexing and querying characteristics as a central operational trade-off.
  • A. Trade-off between Sparse and Dense Vector Indices: The authors recommend sparse encoder indexes for very large enterprises and multi-tenancy with federated search queries.This recommendation addresses the scale-related limitations discussed for large document volumes.
  • B. Blended Retrievers without Metadata: Metadata or other relevant informational facets improve blended-retriever efficacy, whereas metadata-poor datasets such as CoQA show less impressive results.The CoQA results are reported in Table VI.
  • B. Blended Retrievers without Metadata: Without metadata, hybrid queries do not improve over basic queries on CoQA, although Sparse Encoder-based semantic searches outperform traditional methods.The limitation indicates that metadata is critical for complex query structures while sparse semantic search remains favorable.
  • B. Blended Retrievers without Metadata: NDCG@10 for Retriever and F1,EM for RAG are poor proxies for human alignment in Generative Q&A systems.The authors identify better RAG evaluation metrics as a key area for future work.

VII. CONCLUSION

The Blended RAG pipeline is effective across multiple datasets without dataset-specific training or prompt exemplars, supporting robust zeroshot generalization. Blended Search and Blended Retrievers improve retrieval and RAG performance, establishing benchmarks on TREC-COVID, SQUAD, and NQ.

  • Generalization: Blended RAG achieves effectiveness across multiple datasets without specific training or prompt-engineering exemplars, indicating robust zeroshot generalization.The conclusion emphasizes applicability beyond datasets used for training and the absence of few-shot exemplars.
  • Blended Search: 87% Retriever Accuracy on TREC-COVID is achieved by combining Sparse Encoder indices with ‘Best Fields’ queries.The conclusion identifies this Semantic Search construct as superior across all evaluated settings.
  • Blended Retrievers: 68% F1 Score on SQUAD and 42% EM Score on NQ establish new benchmarks for non-tuned Q&A systems.The conclusion attributes these results to the enhancement of RAG through Blended Retrievers.
  • Blended Retrievers: Blended Retrievers improve RAG results beyond focusing on LLM size and type, including with relatively smaller LLMs.The conclusion presents this as a foundation for more intelligent and contextually aware Generative Q&A systems.
Loading 2404.07220v2…