PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation

Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, Jie Zhou

arXiv:2402.18150v2cs.CLcs.AIcs.IR

TL;DR

LLMs can struggle to use retrieved texts of varied quality, including ignoring or being misled by them. INFO-RAG trains them unsupervisedly as information refiners, and experiments report improved RAG performance across datasets and tasks.

  • Problem

    LLM training does not explicitly teach models to use retrieved texts of varied quality, and models may struggle with complex, incomplete, incorrect, or noisy retrievals.

    Source: §1 · pp. 1–2

  • Method

    INFO-RAG unsupervisedly trains LLMs to combine retrieved information and model knowledge to refine outputs across three retrieval scenarios.

    Source: §1 · p. 2

  • Results

    Across 11 datasets in 7 tasks, INFO-RAG improves LLM RAG performance; reported zero-shot results show an average 9.39% improvement in retrieved-knowledge utilization.

    Source: §4.3 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that refinement can support positive information gain by extracting correct information and resisting or rectifying retrieved errors and noise.

    Source: §1 · p. 2

    Experiments cover only 7B- and 13B-parameter models because of limited computing resources.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating additional information from retrieval. However, studies have shown that LLMs still face challenges in effectively using the retrieved information, even ignoring it or being misled by it. The key reason is that the training of LLMs does not clearly make LLMs learn how to utilize input retrieved texts with varied quality. In this paper, we propose a novel perspective that considers the role of LLMs in RAG as ``Information Refiner'', which means that regardless of correctness, completeness, or usefulness of retrieved texts, LLMs can consistently integrate knowledge within the retrieved texts and model parameters to generate the texts that are more concise, accurate, and complete than the retrieved texts. To this end, we propose an information refinement training method named InFO-RAG that optimizes LLMs for RAG in an unsupervised manner. InFO-RAG is low-cost and general across various tasks. Extensive experiments on zero-shot prediction of 11 datasets in diverse tasks including Question Answering, Slot-Filling, Language Modeling, Dialogue, and Code Generation show that InFO-RAG improves the performance of LLaMA2 by an average of 9.39\% relative points. InFO-RAG also shows advantages in in-context learning and robustness of RAG.

1 Introduction

The introduction argues that LLMs are not trained to use retrieved texts of varied quality as evidence, motivating an “Information Refiner” role for RAG. It proposes unsupervised refinement training and reports improvements across several RAG task families.

1 Introduction

  • LLM pre-training minimizes the likelihood of the entire retrieved-text, question, and answer sequence, unlike RAG training, which targets the question and generated answer using retrieved texts as reference.The authors identify this objective mismatch as a reason LLMs are not explicitly trained to use retrieved texts of varied quality.Source: §1 · p. 1
  • Retrieved texts may be difficult to use, incomplete, incorrect, or noisy, while prompt techniques cannot materially change utilization and task-specific fine-tuning can restrict zero-shot generalizability.The paper also notes the annotation cost of constructing data for many RAG tasks.Source: §1 · p. 2
  • The paper frames LLMs as “Information Refiners” that integrate retrieved knowledge with model parameters to produce more concise, accurate, and complete text.This is intended to support positive information gain regardless of retrieved texts’ correctness, completeness, or usefulness.Source: §1 · p. 2
  • INFO-RAG trains unsupervisedly on three retrieval scenarios: complete evidence, incomplete or incorrect evidence, and relevant evidence without an answer.The corresponding tasks extract concise relevant knowledge, verify and correct or complete retrieved knowledge, or answer from model parameters using relevant context.Source: §1 · p. 2
  • INFO-RAG improves zero-shot RAG performance across Question Answering, Slot-Filling, Language Modeling, Dialogue, and Code Generation, with reported advantages in in-context learning and robustness.The paper describes the method as low-cost and general across tasks.Source: §1 · p. 2

2 Related Work

Related work covers retrieval-augmented generation, methods addressing noisy retrieval, and unsupervised training of retrievers and language models. The paper distinguishes its approach by defining an information-refinement role for LLMs in RAG.

Retrieval Augmented Generation

  • RAG supplies language models with external information for generation and is used in open-domain QA, dialogue, and code generation.The cited examples include Lewis et al. (2020), Guu et al. (2020), Karpukhin et al. (2020), and Parvez et al. (2021).Source: §2 · p. 3
  • Prior studies address noise that can interfere with or mislead language models, while this paper argues that they do not clearly define the LLM’s role in RAG.The paper presents “Information Refiner” as its alternative perspective.Source: §2 · p. 3

Unsupervised Learning of RAG

  • Unsupervised RAG learning includes retriever training and language-model training, but cited language-model approaches focus on encoder-decoder architectures rather than mainstream decoder-only LLMs.Examples include REALM, REPLUG, RETRO, and Atlas.Source: §2 · p. 3
  • The paper positions its own unsupervised language-model training as information refinement that improves RAG’s information bottleneck and robustness to retrieved texts.It contrasts this focus with prior unsupervised methods that do not consider the specific role language models should play in RAG.Source: §2 · p. 3

3 Our INFO-RAG

INFO-RAG treats the LLM as an “Information Refiner” and trains it to produce positive information gain across varied retrieved-text scenarios. Its unsupervised training uses scenario-specific tasks and Wikipedia-derived samples.

3.1 Positive Information Gain in RAG

  • Positive information gain means extracting concise correct content, verifying and repairing incomplete or incorrect content, or answering from model knowledge using semantically relevant context.Source: §3.1 · p. 3

3.2.1 Data Collection

  • Training samples come from English Wikipedia: a sentence is split into a prefix and target, while other sentences simulate retrieved texts across scenarios.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.2 Data Construction and Training Tasks

  • For Scenario 1, Select and Copy conditions on all sentences in S, requiring selection of prefix-matching text and copying its continuation.Source: §3.2.2 · p. 4
  • For Scenario 2, Correct and Complete corrupts 30% of informative tokens with masking or substitutions, then trains recovery of the target knowledge.Source: §3.2.2 · pp. 4–5
  • For Scenario 3, Contextual Stimulation removes the target sentence, leaving semantically relevant texts that cue generation from model knowledge.Source: §3.2.2 · p. 5

3.2.3 Training Strategy

  • INFO-RAG mixes the three tasks for LoRA training, alternating batches at 20% Select and Copy and 40% each for the other tasks.The trained LoRA parameters can be loaded for RAG and unloaded when RAG is not needed.Source: §3.2.3 · p. 5

4 Experiments

Across eleven datasets and seven tasks, INFO-RAG improves zero-shot RAG performance and shows gains across short-answer, reasoning, long-form, dialogue, language-modeling, and code-generation tasks. The experiments also examine in-context learning, stronger RAG frameworks, retrieval robustness, and the effects of training tasks.

4.1 Datasets and Evaluation Metrics

  • Evaluation spans eleven datasets across seven tasks, including open-domain and multi-hop QA, slot filling, long-form QA, dialogue, language modeling, and code generation.Metrics include exact match or accuracy, ROUGE-L, F1, and CodeBLEU, depending on the task.Source: §4.1 · pp. 5–6

4.2 Experimental Settings

  • Experiments fine-tune LLaMA-2 7B and 13B models and chat variants with LoRA; retrieval uses shared documents across baselines and INFO-RAG within each experiment.For several tasks the setup uses top-five retrieved passages; other tasks use dataset-provided contextual passages.Source: §4.2 · p. 6

4.3 Experimental Results

  • 9.39% average improvement is reported for INFO-RAG on zero-shot retrieval-augmented generation across 11 datasets and 7 tasks.Table 1 presents the overall zero-shot results.Source: §4.3 · p. 6, Table 1 · p. 6
  • INFO-RAG improves RAG performance on short-answer open-domain QA and slot-filling, multi-hop reasoning, long-form QA, dialogue, language modeling, and code generation.The authors report gains for tasks requiring cross-passage reasoning and for code generation despite training only on natural language.Source: §4.3 · p. 6
  • INFO-RAG can further improve RAG with in-context learning, whereas LLaMA-2 may fail to improve or may be hurt by few-shot examples.The authors attribute this difference to INFO-RAG helping LLaMA understand the task form of RAG and learn general task patterns from examples.Source: §4.3 · p. 7, Table 3 · p. 7
  • INFO-RAG-trained SearChain performs better, supporting its use alongside multi-step retrieval methods in the open-retrieval setting.The comparison uses SearChain with LLaMA-2-13B-chat; Table 4 covers enhancement to prior SOTA frameworks.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Analysis

  • INFO-RAG shows advantages across correct-answer-present, answer-replaced, and no-answer retrieval scenarios, with the analysis reporting robustness across these conditions.The no-answer scenario is described as more difficult than the answer-replaced scenario.Source: §4.4 · pp. 7–8, Table 2 · p. 7
  • Wikipedia language-model training alone adds no improvement over LLaMA-2; S2 and S3 gain in their scenarios, while S1 alone hurts but all three tasks together perform best.The authors attribute S1's standalone effect to overfitting and say adding S2 and S3 helps the model learn the information-refinement task paradigm.Source: §4.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5 Conclusion

INFO-RAG treats LLMs in RAG as information refiners, integrating retrieved text with model knowledge to produce more concise, accurate, and complete text. The unsupervised method improves zero-shot RAG performance across diverse tasks and also shows advantages in in-context learning and robustness.

5 Conclusion

  • INFO-RAG trains LLMs to integrate retrieved texts with model knowledge, producing text that is more concise, accurate, and complete.The proposed information-refinement approach applies regardless of retrieved text correctness, completeness, or usefulness.Source: §5 · p. 8
  • Across 11 datasets in 7 tasks, INFO-RAG improves zero-shot RAG performance and also shows advantages in in-context learning and RAG robustness.The paper also reports that it can be combined with a state-of-the-art RAG framework to further improve performance.Source: §5 · p. 9
  • INFO-RAG is an unsupervised, low-cost training method intended to generalize across various tasks.Source: §5 · p. 9

Limitations

The experiments cover only models with 7B and 13B parameters because of limited computing resources.

Limitations

  • Limited computing resources restricted experiments to 7B and 13B models, leaving larger model sizes for future exploration.Source: Limitations · p. 9

Ethics Statement

The authors state that the paper introduces no additional ethical concerns.

Ethics Statement

A.1 Robustness to ratio of Positive Passages

INFO-RAG is reported to perform better at low positive-passage ratios and to be more stable than the baseline as the ratio changes from 100% to 0%.

A.1 Robustness to ratio of Positive Passages

  • INFO-RAG performs better at low positive-passage ratios and is more stable than the baseline as the ratio changes from 100% to 0% (Max ∆).Source: §A.1 · p. 11
  • The experiment simulates changes in retrieval performance by varying positive and negative passages in the retrieved list, using LLaMA-2-13B-chat.Source: §A.1 · p. 11

A.2 Robustness to Positive Passage Position

INFO-RAG is reported to outperform LLaMA-2 across every tested position of the positive passage and to show more stable performance as that position changes.

A.2 Robustness to Positive Passage Position

  • INFO-RAG outperforms LLaMA-2 at every tested position of the positive passage in the retrieved list.The experiment varied the positive passage's position from 0 to 9 in lists with a 1:9 positive-to-negative passage ratio, using LLaMA-2-13B-chat.Source: §A.2 · p. 11
  • INFO-RAG's performance varies more stably than the baseline as the positive passage changes position.The passage reports a more stable performance profile across positions, characterized as a lower Max ∆, without giving its value.Source: §A.2 · p. 11
  • The position-robustness evaluation used a retrieved list mixing positive and negative passages at a 1:9 ratio.The positive passage position was varied from 0 to 9; the evaluated base model was LLaMA-2-13B-chat.Source: §A.2 · p. 11

A.3 Robustness to Number of Retrieved Passages

Across several robustness and ablation settings, INFO-RAG outperforms the LLaMA-2 baseline, including with varying retrieval-set sizes and a BM25 retriever. Its masking strategy is especially effective when retrieved answers are replaced or absent, while its MMLU performance remains close to the original LLaMA-2.

A.3 Robustness to Number of Retrieved Passages

  • INFO-RAG consistently outperforms the baseline as the number of retrieved passages varies from 1 to 10.The experiment uses LLaMA-2-13B-chat and evaluates performance across different retrieval-set sizes.Source: §A.3 · p. 11, Table 10 · p. 13

A.4 Ablation Study on Masking Strategy

  • The proposed masking strategy is more effective than simple masking, especially when answers are replaced or retrieval finds no answer.The comparison covers three scenarios: correct answers present, answers replaced, and no answers retrieved.Source: §A.4 · p. 12, Table 13 · p. 12, Table 12 · p. 13

A.5 Works with Different Retriever

  • INFO-RAG performs better than the baseline when BM25 is used as the retriever.The comparison is identified as Table 11, which is captioned as results for works based on BM25.Source: §A.5 · p. 12, Table 11 · p. 12

A.6 Performance on MMLU

  • On MMLU without RAG, INFO-RAG performs close to the original LLaMA-2, indicating that its basic language-understanding ability is not damaged.The paper also reports maintaining versatility and avoiding catastrophic forgetting; its prefix language-modeling training paradigm is consistent with LLM pre-training while learning information refinement for next-token prediction.Source: §A.6 · pp. 12–13, Table 14 · p. 13
Loading 2402.18150v2…