PaperlayerView exact revision on arXiv

Source-linked AI summary

REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering

Yuhao Wang, Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, Ji-Rong Wen

arXiv:2402.17497v2cs.CLcs.IR

TL;DR

RAG uses retrieved documents to extend LLM knowledge, but existing methods may not assess their relevance precisely, risking misleading use of external knowledge. REAR combines explicit relevance assessment with bi-granularity and noise-resistant training. The paper reports stronger QA performance across four datasets and identifies broader RAG tasks as an area still requiring evaluation.

  • Problem

    Existing methods do not precisely assess retrieved-document relevance, limiting reliable use of external knowledge in RAG.

    Source: §1 · pp. 1–2

  • Method

    REAR incorporates an explicit document-relevance assessment module into LLM generation and combines bi-granularity relevance fusion with noise-resistant training.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    REAR surpasses the baselines in QA performance on trained and non-training datasets, and experiments on four datasets demonstrate effectiveness and generalization.

    Source: §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    REAR’s relevance signals guide generation toward effective use of parametric and external knowledge.

    Source: §5.2 · p. 7

    Applicability to broader RAG tasks, including those in the KILT benchmark, remains to be thoroughly evaluated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Considering the limited internal parametric knowledge, retrieval-augmented generation (RAG) has been widely used to extend the knowledge scope of large language models (LLMs). Despite the extensive efforts on RAG research, in existing methods, LLMs cannot precisely assess the relevance of retrieved documents, thus likely leading to misleading or even incorrect utilization of external knowledge (eg., retrieved documents). To address this issue, in this paper, we propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA). As the key motivation, we aim to enhance the self-awareness regarding the reliability of external knowledge for LLMs, so as to adaptively utilize external knowledge in RAG systems. Specially, we develop a novel architecture for LLM-based RAG systems, by incorporating a specially designed assessment module that precisely assesses the relevance of retrieved documents. Furthermore, we propose an improved training method based on bi-granularity relevance fusion and noise-resistant training. By combining the improvements in both architecture and training, our proposed REAR can better utilize external knowledge by effectively perceiving the relevance of retrieved documents. Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches. Our codes can be accessed at https://github.com/RUCAIBox/REAR.

1 Introduction

RAG extends LLM knowledge with retrieved documents, but irrelevant or weakly assessed documents can mislead answers. REAR adds explicit relevance assessment and training strategies to guide external-knowledge use, with reported gains on open-domain QA benchmarks and out-of-domain evaluation.

1 Introduction

  • Existing RAG systems can be misled by irrelevant retrieved documents, and adding multiple references can amplify noise while LLMs struggle to filter documents and integrate internal knowledge.Source: §1 · p. 1
  • Prior robustness approaches use binary relevance labels, which are sparse and do not capture fine-grained relevance or varied degrees of document relevance.Source: §1 · p. 2
  • REAR adds explicit relevance-assessment modules to guide LLM generation, with bi-granularity relevance fusion and noise-resistant training to improve document discrimination.The assessment signals are fed back to reduce distraction from irrelevant external knowledge.Source: §1 · p. 2
  • Experiments on public open-domain QA benchmarks support REAR’s effectiveness, and out-of-domain evaluation on multiple benchmarks demonstrates its reported generalization capability.Source: §1 · p. 2

2 Related Work

Related work spans retrieval-reader QA systems and methods for improving retrieval-augmented LLMs. REAR addresses the remaining challenge of assessing document relevance precisely enough to reduce interference from irrelevant content.

2 Related Work

  • Traditional IR and neural reading-comprehension systems retrieve evidence for extractive or generative readers, while systems such as REALM, RAG, RETRO, and In-context RALM improve factual generation but lack explicit relevance discernment.Source: §2 · p. 2
  • Retrieval-augmented LLM research includes retriever alignment, joint retriever–language-model training, expanded knowledge sources, and query rewriting.Source: §2 · p. 2
  • Self-RAG, Luo et al. (2023), and RobustLM make an initial relevance judgment, but these approaches still lack accurate relevance discrimination and remain vulnerable to irrelevant content.Source: §2 · p. 2

3 Task Formulation

The task is open-domain question answering: retrieve evidence from a large document collection and generate answers to questions. REAR focuses on assessing query–document relevance and using that signal to reduce noise during generation.

3 Task Formulation

  • Open-domain QA answers questions using a large document collection, typically through a retriever that finds relevant evidence and a reader that generates an answer from it.Source: §3 · p. 2
  • Given query q, the retriever returns top-k documents D, and the LLM generates an answer for each reference document to form an answer set A.Source: §3 · pp. 2–3
  • A final answer is selected from A to align best with q, while the formulation highlights precise query–document relevance assessment and relevance signals for noise-resistant generation.Source: §3 · p. 3

4 Methodology

REAR adds explicit relevance assessment to RAG and uses relevance signals to guide answer generation and verify answer reliability. Its training combines coarse and fine-grained relevance supervision with noise-resistant training, supported by data construction methods that avoid GPT APIs.

4.1.1 Relevance Assessment

  • REAR maps each query-document pair to a relevance embedding, which an assessment module implemented as a linear projection converts into a relevance score.The framework describes this explicit assessment as part of its relevance-aware RAG architecture.Source: §4.1.1 · p. 3, §4.1 · p. 3

4.1.2 Relevance-guided Generation

  • The relevance score is embedded as a dense vector that guides answer generation toward internal knowledge when relevance is low and external evidence when it is high.The scalar score may not be fully utilized directly by the LLM.Source: §4.1.2 · p. 4

4.1.3 Knowledge Reliability Verification

  • Answer verification considers both whether the document is reliable and whether the LLM stands by its answer without the document.The paper names these strategies source-reliability and knowledge-consistency, then combines the consistency and relevance scores to select the final answer.Source: §4.1.3 · p. 4

4.2.1 Bi-granularity Relevance Fusion

  • Bi-granularity relevance fusion combines coarse binary supervision with preference constraints derived from estimated relevance scores.The paper motivates fine-grained preferences because binary labels are sparse and do not capture relevance degrees precisely.Source: §4.2.1 · p. 4

4.2.2 Noise-resistant Training

  • Noise-resistant training incorporates negative documents so the LLM can discern irrelevant evidence without being encumbered by it.The overall REAR loss combines the bi-granularity and noise-resistant losses.Source: §4.2.2 · p. 4, §4.2.3 · p. 5

4.2.3 Training Data Construction

  • Training-data construction uses a small cross-encoder reranker for continuous relevance scores and combines them with binary labels, without GPT APIs.Its refined SimANS sampling uses a decay scaler when relevance scores are high to reduce the chance of sampling false negatives.Source: §4.2.3 · pp. 4–5

4.3 Discussion

  • REAR’s architecture supports fine-grained relevance signals, consistency checks, and the proposed losses; the paper reports more precise relevance evaluation and better generation performance.The efficiency discussion attributes reduced inference time versus Self-RAG primarily to PagedAttention preserving relevance-assessment calculations, and contrasts CoN’s extensive-paragraph processing and quadratic input-length time complexity.Source: §4.3 · p. 5, Table 2 · p. 5

5 Experiments

Experiments evaluate REAR on four open-domain QA datasets and analyze its relevance assessment, training components, and robustness to retrieved-document conditions. The reported results favor REAR over baselines and show gains across several relevance and noise settings.

5.2 Main Results

  • REAR surpasses all baselines in QA performance across NQ, TriviaQA, WebQ, and SQuAD, including the averaged results.Self-RAG† uses Acc rather than the stricter EM metric.Source: Table 3 · p. 6, §5.2 · p. 7

5.1 Experimental Setup

  • On four datasets, REAR is evaluated against prompting and RAG-specific baselines using QA accuracy and document-relevance metrics.The setup uses NQ training data and tests generalization on TriviaQA, WebQ, and SQuAD; metrics include EM, F1, Hit@1, and JAcc.Source: §5.1 · p. 6

5.2 Main Results

  • The results indicate that REAR’s data-construction strategy is effective and less costly, without access to GPT APIs.Self-RAG uses GPT-4 for relevance labels, while RobustLM and REAR use the proposed label machine and sampling method.Source: §5.2 · p. 7
  • REAR significantly enhances relevance discrimination compared with fine-tuned generative approaches, which adequately discriminate relevance but struggle to determine its degree.The comparison is reported in Table 4.Source: §5.2 · p. 7, Table 4 · p. 7

5.3.1 Ablation Study

  • Removing the assessment module reduces comparison accuracy, while removing knowledge consistency improves Hit@1 but lowers EM and F1 relative to knowledge consistency.The assessment design is described as minimizing interference between generation and relevance discrimination and facilitating varied loss functions.Source: §5.3.1 · p. 7
  • Removing bi-granularity relevance training weakens relevance comparison and performance, while removing noise-resistant training causes a notable decline in generation against irrelevant-document interference.The ablation variants replace bi-granularity fusion with coarse-grained loss and exclude gold-noise data pairing, respectively.Source: §5.3.1 · pp. 7–8

5.3.2 Impact of Retrieved Documents

  • With only the top retrieved document, REAR generates more accurate responses than other LLMs when that document is irrelevant.The comparison uses factual-generation accuracy on NQ and also reports that fine-tuned Self-RAG and REAR perform well with relevant documents.Source: §5.3.2 · p. 8, Table 6 · p. 8
  • REAR performs well with one document and remains robust across retriever capacities, with a large improvement over other baselines when using BM25.Figure 3 compares document counts and retriever engines; R1, R2, and R3 denote BM25, Contriever-msmarco, and the FiD-distilled retriever.Source: §5.3.2 · p. 8, Figure 3 · p. 8

6 Conclusion

REAR combines an explicit relevance-assessment module with bi-granularity and noise-resistant training to guide external-knowledge use. Experiments on four datasets demonstrate effectiveness and generalization, while extending source-level granularity and task coverage remains future work.

6 Conclusion

  • REAR combines an explicit assessment module with bi-granularity relevance fusion and noise-resistant training to support fine-grained assessment and adaptive use of retrieved documents.Its data-construction strategy collects high-quality data without GPT APIs.Source: §6 · p. 8
  • Future work will extend REAR to passage- or sentence-level source utilization and to other knowledge-intensive tasks.Source: §6 · p. 8

Limitations

REAR assesses retrieved documents and dynamically uses them, but its granularity and evaluated task scope remain limited.

Limitations

  • The approach assesses relevance at the document level and dynamically utilizes retrieved documents, while reporting improved factual accuracy.It does not guide the LLM to focus more granularly on key sentences or tokens within documents.Source: Limitations · p. 9
  • The stated granularity boundary is sentence- or token-level focus within retrieved documents.Source: Limitations · p. 9
  • REAR’s applicability to broader RAG tasks, including those in the KILT benchmark, remains to be thoroughly evaluated.Source: Limitations · p. 9

A Details on Fine-Gained Relevance Optimization

This section explains why fine-grained relevance matters and describes training strategies intended to reduce the impact of reranker annotation errors.

A Details on Fine-Gained Relevance Optimization

  • Binary labels mark both D1 and D2 relevant despite D1 allowing direct answer derivation and D2 requiring additional external knowledge for induction.Training only on binary classification may fail to distinguish their relative relevance and lead to inaccurate finer-grained judgments.Source: A Details on Fine-Gained Relevance… · p. 13
  • RocketQAv2 cross-encoder reranker scores provide a lower-cost alternative to distilling GPT-4 ranking results, but can still be wrong.Source: A Details on Fine-Gained Relevance… · p. 13
  • Training reduces annotation-noise effects through sampling that lowers false-negative selection, linear fusion of binary labels and cross-encoder scores, and ignoring score differences below 0.1.The passage states these strategies enhance training-data quality and REAR performance.Source: A Details on Fine-Gained Relevance… · p. 13

B Details on Dataset.

The evaluation uses four open-domain QA datasets, with NQ serving as the training and inference dataset and the other three used only for inference.

B Details on Dataset.

  • The four datasets are Natural Questions (NQ), TriviaQA, WebQuestions (WQ), and SQuAD.Source: B Details on Dataset · p. 13
  • NQ questions come from Google search queries and have human-annotated answer spans in Wikipedia; its training set contains 58,880 samples.NQ is used for both training and inference, while TriviaQA, WQ, and SQuAD are used only for inference.Source: B Details on Dataset · p. 13
  • TriviaQA pairs trivia questions with answers initially extracted from online sources, while WQ uses Google Suggest questions and Freebase entities as answers.Source: B Details on Dataset · p. 13
  • SQuAD evaluates reading comprehension and is also used to train and test open-domain QA engines.Source: B Details on Dataset · p. 13

C Details on Document Collection

The section describes dataset-specific retrievers and identifies the reported retrieval metrics and direct-RAG prompt figure.

C Details on Document Collection

  • Inference uses FiD-distilled retrievers for NQ and TriviaQA, and a Contriever-msmarco-based strategy with in-batch negatives and joint retriever-ranker training for SQuAD and WQ.Source: C Details on Document Collection · p. 13
  • Table 8 reports test-set hit or recall rates for the retrievers used for testing across datasets.Source: Table 8 · p. 13
  • Table 9 compares three NQ test-set retrievers using Hit@10 and MRR@10; R1, R2, and R3 denote BM25, Contriever-msmarco, and a FiD-reader-attention-distilled dense retriever.Hit@10 measures correct answers within the top 10, while MRR@10 reflects the rank of the first correct answer within the top 10.Source: Table 9 · p. 14
  • Figure 5 presents prompts for direct RAG QA.Source: Figure 5 · p. 14

D Details on Implementation

The implementation jointly trains relevance assessment and relevance-guided generation, while baseline details specify prompt-based inference and answer behavior. The experiments use a one-epoch schedule on eight A100 GPUs.

D Details on Implementation

E Details on Baselines

Related papers

Loading 2402.17497v2…