PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, Jong C. Park

arXiv:2403.14403v2cs.CLcs.AI

TL;DR

Existing retrieval-augmented QA systems can waste computation on simple queries or underperform on complex multi-step questions. Adaptive-RAG trains a smaller classifier with automatically collected labels to select no-retrieval, single-step, or multi-step strategies, and reports improved accuracy and efficiency across varied open-domain QA datasets.

  • Problem

    Existing one-size-fits-all retrieval strategies either impose unnecessary overhead on simple queries or inadequately handle complex multi-step questions.

  • Method

    Adaptive-RAG uses a smaller classifier trained on automatically collected labels to select non-retrieval, single-step, or multi-step QA strategies according to query complexity.

  • Results

    Adaptive-RAG enhances overall QA accuracy and efficiency across open-domain datasets spanning single-hop and multi-hop questions.

  • Takeaways & Limitations

    The framework allocates more resources to complex queries while handling simpler queries efficiently across varying query complexities.

  • Takeaways & Limitations

    Automatically generated complexity labels may be incorrect, and the classifier is limited to three complexity levels.

Abstract

from arXiv · show

Retrieval-Augmented Large Language Models (LLMs), which incorporate the non-parametric knowledge from external knowledge bases into LLMs, have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA). However, even though there are various approaches dealing with queries of different complexities, they either handle simple queries with unnecessary computational overhead or fail to adequately address complex multi-step queries; yet, not all user requests fall into only one of the simple or complex categories. In this work, we propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs from the simplest to the most sophisticated ones based on the query complexity. Also, this selection process is operationalized with a classifier, which is a smaller LM trained to predict the complexity level of incoming queries with automatically collected labels, obtained from actual predicted outcomes of models and inherent inductive biases in datasets. This approach offers a balanced strategy, seamlessly adapting between the iterative and single-step retrieval-augmented LLMs, as well as the no-retrieval methods, in response to a range of query complexities. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems, compared to relevant baselines including the adaptive retrieval approaches. Code is available at: https://github.com/starsuzi/Adaptive-RAG.

1 Introduction

Adaptive-RAG addresses the mismatch between query complexity and fixed retrieval strategies by selecting among no-retrieval, single-step, and multi-step approaches. A classifier enables this selection, and experiments report improved overall accuracy and efficiency across diverse open-domain QA datasets.

  • Retrieval-augmented LLMs add external knowledge to address factual errors caused by reliance on parametric memory.
  • Single-step retrieval can answer many queries, but complex questions may require connecting multiple documents through iterative reasoning.
  • Fixed multi-step processing is costly for simple queries, whereas single-step or non-retrieval strategies can be insufficient for complex queries.
  • Adaptive-RAG uses a smaller classifier to predict query complexity and select among non-retrieval, single-step, and iterative strategies.
  • Experiments on open-domain QA datasets show improved overall accuracy and efficiency across multiple LLMs compared with prior adaptive strategies.

2 Related Work

Related work develops retrieval-augmented systems for single-hop and multi-hop QA, including iterative reasoning and adaptive retrieval. These approaches generally overlook the broad distribution of query complexities and may therefore apply costly iterative processing too broadly.

  • Open-domain QA: Open-domain QA combines document retrieval with answer interpretation through retriever and reader modules.
  • Multi-hop QA: Multi-hop QA requires gathering and contextualizing information from multiple documents, often through iterative retrieval and reasoning.
  • Multi-hop QA: Recent methods interleave reasoning with retrieval or repeatedly retrieve documents based on generated-token confidence.
  • Multi-hop QA: Iterative retrieval can be inefficient when applied to every query, because some real-world questions are simple enough for less elaborate strategies.
  • Adaptive Retrieval: Adaptive retrieval methods often make a binary retrieve-or-not decision, which may not support complex queries requiring multiple reasoning steps.

3 Method

Adaptive-RAG first defines non-retrieval, single-step, and multi-step QA strategies, then predicts query complexity to select among them. Its classifier is trained from automatically constructed labels derived from model outcomes and dataset biases.

  • Preliminaries: Non-retrieval QA generates an answer directly from the user query but is problematic when precise external knowledge is required.
  • Preliminaries: Single-step QA retrieves relevant documents from an external knowledge source and supplies them with the query to the LLM.
  • Preliminaries: Multi-step QA repeatedly retrieves documents and incorporates prior documents and outcomes as context while progressively refining the answer.
  • Adaptive-RAG: Adaptive-RAG determines query complexity before solving the query and dynamically selects a strategy spanning non-retrieval, single-step, and multi-step processing.
  • Query Complexity Assessment: A smaller classifier assigns three complexity levels: A for direct LLM answers, B when single-step retrieval is needed, and C for the most extensive solution.
  • Training Strategy: Because no annotated query-complexity dataset exists, training data is automatically constructed from model prediction outcomes and inductive biases in benchmark datasets.
  • Adaptive-RAG: The method considers three query-complexity levels and leaves finer-grained complexity distinctions for future work.

4 Experimental Setups

The experiments evaluate Adaptive-RAG on unified single-hop and multi-hop open-domain QA, comparing effectiveness and efficiency across retrieval strategies, models, and datasets. The setup uses benchmark datasets, shared retrieval components, and classifier training data constructed without overlap with QA test queries.

  • Datasets: The evaluation combines single-hop and multi-hop QA datasets to simulate queries with varying complexities.Single-hop datasets include SQuAD v1.1, Natural Questions, and TriviaQA; multi-hop datasets include MuSiQue, HotpotQA, and 2WikiMultiHopQA.
  • Compared Methods: The comparison covers Simple, Adaptive, and Complex strategies, including No Retrieval, Single-step, Adaptive Retrieval, Self-RAG, Adaptive-RAG, and Multi-step approaches.Adaptive-RAG adaptively performs retrieval based on question complexity.
  • Metrics: Effectiveness is measured with F1, EM, and Accuracy, while efficiency is assessed with two additional metrics.The evaluation explicitly considers task performance, efficiency, and their trade-offs.
  • Models and Retrieval: The experiments use BM25 as the shared sparse retriever and FLAN-T5-XL, FLAN-T5-XXL, and GPT-3.5 as answer-generating LLMs.FLAN-T5-XL has 3B parameters and FLAN-T5-XXL has 11B parameters.
  • Classifier Training: Adaptive-RAG trains a T5-Large query-complexity classifier using automatically labeled queries, with classifier-training queries separated from QA test queries.Labels are derived from dataset inductive biases and predicted outcomes of retrieval strategies; 400 queries are sampled from six datasets for each labeling process.

5 Experimental Results and Analyses

Experiments show that Adaptive-RAG improves the accuracy and efficiency of retrieval-augmented QA by selecting strategies according to query complexity. Analyses examine adaptive performance, classifier behavior, efficiency, training data, and qualitative handling of simple and complex questions.

  • Main Results: Adaptive-RAG is more effective than competing adaptive strategies and more efficient than complex multi-step retrieval across model sizes.The comparison is reported over considered datasets, with efficiency motivated by the cost of accessing LLMs.
  • Classifier Performance: The classifier distinguishes no-retrieval, single-retrieval, and multi-retrieval queries, supporting more fine-grained handling than binary retrieval decisions.Classifier accuracy is reported as better than competing adaptive retrieval baselines.
  • Classifier Performance: 31% of ‘C (Multi)’ queries are misclassified as ‘B (One)’, while 23% of ‘B (One)’ queries are misclassified as ‘C (Multi)’.Other reported confusions include 47% of ‘A (No)’ as ‘B (One)’ and 22% as ‘C (Multi)’.
  • Efficiency Analysis: Exact per-query timing shows that identifying simple or straightforward queries can substantially improve efficiency.Table 3 also reports the distribution of predicted complexity labels.
  • Training Data Analysis: Training with silver predictions and dataset inductive bias improves classifier-supported QA compared with using either annotation source alone.The combined strategy better predicts multi-hop queries, while silver-only labels may generalize less effectively.
  • Case Study: Adaptive-RAG avoids retrieval for a simple question but retrieves details absent from parametric memory for a complex question.The case study contrasts this behavior with unnecessary or insufficient retrieval by Adaptive Retrieval.

6 Conclusion

Adaptive-RAG dynamically selects among non-retrieval, single-step, and multi-step retrieval strategies according to query complexity. On open-domain QA datasets, it improves overall accuracy and efficiency while allocating more resources to complex queries.

  • 6 Conclusion: Adaptive-RAG dynamically adjusts between non-retrieval, single-step, and multi-step retrieval strategies based on query complexity.A smaller language model classifier predicts query complexity from automatically annotated query-complexity pairs.
  • 6 Conclusion: Adaptive-RAG improves overall QA accuracy and efficiency across datasets containing single-hop and multi-hop questions.The framework is compared with existing one-size-fits-all approaches that are either minimalist or maximalist across query complexities.
  • 6 Conclusion: The framework allocates more resources to complex queries while efficiently handling simpler queries.This contrasts with approaches that apply the same retrieval strategy regardless of query complexity.

Limitations

The classifier remains an initial system component with opportunities for improvement in both its training data and architecture. Automatically generated complexity labels may be incorrect, and the current classifier trails an ideal oracle classifier.

  • Limitations: Automatically generated query-complexity labels may be incorrect because the labeling process is only one instantiation of complexity annotation.The labels are derived from model prediction outcomes and inductive biases in datasets.
  • Limitations: The current classifier has an effectiveness gap relative to the ideal classifier equipped with an oracle.The paper identifies this gap as evidence that classifier effectiveness can still improve.
  • Limitations: Future work could develop datasets with diverse query-complexity annotations and improve the smaller-LM classifier architecture.The authors describe the current smaller-LM design as an initial, simplest instantiation.

Ethics Statement

Adaptive-RAG is evaluated in realistic settings with diverse user queries, but offensive or harmful inputs may lead to inappropriate retrievals and responses. The authors identify content detection and management as future work.

  • Ethics Statement: Offensive or harmful user inputs may lead to retrieving offensive documents and generating inappropriate responses.The concern applies to both user inputs and documents retrieved within the retrieval-augmented framework.
  • Ethics Statement: Future work should develop methods to detect and manage offensive or inappropriate content in user inputs and retrieved documents.The authors describe this as a critical area for future work.

A.1 Datasets

The evaluation uses publicly available open-domain QA datasets spanning single-hop and multi-hop questions, alongside several retrieval strategies and an Adaptive-RAG oracle comparison. Experiments use FLAN-T5-XXL and A100 GPUs, with a single run because evaluation is costly.

  • A.1 Datasets: The dataset suite includes SQuAD v1.1, Natural Questions, and TriviaQA among publicly available single-hop and multi-hop QA datasets.The datasets originate from document-based annotation, real Google Search queries, and trivia websites, respectively.
  • A.2 Models: Figure 5 reports QA performance using F1 and efficiency using Time/Query for different retrieval-augmented generation approaches.The base language model is FLAN-T5-XXL (11B).
  • A.2 Models: The compared models include no retrieval, single-step retrieval, adaptive retrieval, multi-step retrieval, Adaptive-RAG, and Adaptive-RAG with an oracle classifier.Adaptive-RAG selects among non-retrieval, single-step, and multi-step approaches without architectural changes.
  • A.1 Datasets: Experiments use A100 GPUs with 80GB memory and a single run because retrieval-augmented generation evaluation is costly.The implementations use PyTorch and the Transformers library.

B Additional Experimental Results

Additional experiments compare Adaptive-RAG with other retrieval-augmented generation approaches across FLAN-T5 and GPT-3.5 settings. The reported comparisons emphasize performance and efficiency trade-offs, while the multi-step approach uses IRCoT.

  • Performance vs Time: Adaptive-RAG is reported as significantly more effective and efficient than other retrieval-augmented generation approaches in FLAN-T5-XL and FLAN-T5-XXL comparisons.These comparisons are presented in Figures 4 and 5 as performance–efficiency trade-offs.
  • The multi-step approach uses the state-of-the-art question-answering strategy from IRCoT.
  • Table 7 reports results across a collection of datasets using FLAN-T5-XXL (11B) as the language model.The authors emphasize their results in bold.
  • Table 8 reports results across a collection of datasets using GPT-3.5 Turbo as the language model.The authors emphasize their results in bold.
Loading 2403.14403v2…