PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi

arXiv:2310.11511v1cs.CLcs.AIcs.LG

TL;DR

LLMs remain vulnerable to factual errors, while conventional RAG can retrieve unnecessary or irrelevant passages and fail to ensure supported outputs. SELF-RAG trains one LM to retrieve on demand and reflect on passages and generations using special tokens, enabling controllable decoding. Across six tasks, it significantly outperforms larger LLMs and conventional retrieval-augmented approaches, although its outputs can still lack complete citation support.

  • Problem

    LLMs produce factual errors, while conventional RAG indiscriminately retrieves passages and does not guarantee that outputs are consistent with relevant evidence.

  • Method

    SELF-RAG trains an LM to retrieve, generate, and critique passages and its own generations using reflection tokens, with retrieval and decoding customizable at inference.

  • Results

    SELF-RAG significantly outperforms LLMs with more parameters and conventional retrieval-augmented approaches across six tasks using multiple metrics.

  • Takeaways & Limitations

    Reflection tokens provide a mechanism for tailoring retrieval and generation behavior to different task requirements while improving quality, factuality, and citation accuracy.

  • Takeaways & Limitations

    SELF-RAG can still generate outputs that are not fully supported by their citations.

Abstract

from arXiv · show

Despite their remarkable capabilities, large language models (LLMs) often produce responses containing factual inaccuracies due to their sole reliance on the parametric knowledge they encapsulate. Retrieval-Augmented Generation (RAG), an ad hoc approach that augments LMs with retrieval of relevant knowledge, decreases such issues. However, indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation. We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's quality and factuality through retrieval and self-reflection. Our framework trains a single arbitrary LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection tokens. Generating reflection tokens makes the LM controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. Experiments show that Self-RAG (7B and 13B parameters) significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks. Specifically, Self-RAG outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA, reasoning and fact verification tasks, and it shows significant gains in improving factuality and citation accuracy for long-form generations relative to these models.

1 INTRODUCTION

LLMs and conventional RAG can produce factual or low-quality outputs because retrieval is indiscriminate and generation is not checked against retrieved evidence. SELF-RAG addresses this by adaptively retrieving, generating, and self-reflecting with controllable reflection tokens, outperforming major baselines across diverse tasks.

  • Conventional RAG retrieves a fixed number of passages regardless of necessity, potentially reducing versatility or introducing irrelevant passages that harm generation quality.
  • SELF-RAG trains a single LM to generate text and reflection tokens that indicate when retrieval is needed and assess retrieved passages and generated outputs.
  • Reflection tokens enable inference-time control over retrieval frequency and model behavior through customizable decoding.
  • SELF-RAG significantly outperforms pre-trained and instruction-tuned LLMs and widely used RAG approaches across six tasks, with higher citation accuracy.

2 RELATED WORK

Prior RAG research improves knowledge-intensive performance through retrieved passages, but commonly relies on fixed or limited retrieval strategies and can incur runtime costs. Related work also explores adaptive retrieval, joint retriever-LM training, and critique-based learning.

  • RAG augments LM inputs with retrieved passages and improves knowledge-intensive tasks after fine-tuning or when used with off-the-shelf LMs.
  • Prior approaches often retrieve a fixed number of passages once, while some methods adapt retrieval or train retrievers and LMs jointly.
  • Adaptive retrieval approaches can improve task performance at the expense of runtime efficiency.
  • Concurrent work fine-tunes retrievers and LMs, whereas SELF-RAG emphasizes retrieval on demand, fine-grained self-reflection, applicability, robustness, and controllability.
  • Unlike PPO-based RLHF, SELF-RAG computes critique offline and inserts it into training data for standard language-model training, reducing training costs.

3 SELF-RAG: LEARNING TO RETRIEVE, GENERATE AND CRITIQUE

SELF-RAG trains an LM to retrieve passages when useful, generate responses, and critique passages and outputs with reflection tokens. Its inference procedure uses adaptive retrieval and critique-guided decoding to customize generation behavior.

  • Framework overview: SELF-RAG uses reflection tokens to signal retrieval needs and evaluate passage relevance, response support, and response usefulness.
  • Problem formalization and overview: The model generates outputs as multiple segments containing both ordinary vocabulary tokens and reflection tokens.
  • Inference overview: At inference, SELF-RAG predicts whether retrieval is useful; without retrieval it continues as a standard LM, while retrieval triggers passage critique, generation, and support evaluation.
  • Training overview: The generator is trained on retrieved passages and critic-produced reflection tokens using next-token prediction over an expanded vocabulary.
  • Training the critic model: The critic is trained on supervised reflection-token data, with GPT-4 used to generate feedback that is distilled into an in-house critic model.
  • Training the critic model: The critic achieves higher than 90% agreement with GPT-4-based predictions on most reflection-token categories.
  • Adaptive retrieval: Retrieval is triggered dynamically or when the normalized probability of Retrieve=Yes exceeds a chosen threshold.
  • Tree-decoding with critique tokens: When retrieval occurs, segment-level beam search scores candidate continuations using weighted critique-token probabilities, whose weights can be adjusted at inference.

4 EXPERIMENTS

The experiments evaluate SELF-RAG and diverse baselines across six tasks, using zero-shot instructions and metrics covering correctness, factuality, fluency, and citation quality.

  • Settings: Experiments use zero-shot task instructions without few-shot demonstrations and train on 150k instruction-output pairs.The generator uses Llama2 7B or 13B, while the critic uses Llama2 7B.
  • Evaluation tasks: SELF-RAG is evaluated across closed-set verification and reasoning, short-form open-domain QA, and long-form biography and QA generation tasks.The tasks include PubHealth, ARC-Challenge, PopQA, TriviaQA-unfiltered, biography generation, and ALCE-ASQA.
  • Metrics: Evaluation uses accuracy for closed-set tasks, while generation tasks use correctness, factuality, fluency, and citation precision and recall metrics.FactScore evaluates biographies; ASQA uses str-em, MAUVE, citation precision, and citation recall.
  • Baselines: The study compares SELF-RAG with pretrained, instruction-tuned, proprietary, and retrieval-augmented language-model baselines.Baselines include Llama2, Alpaca, ChatGPT, Llama2-chat, standard RAG, and models fine-tuned with or without retrieval.
  • Settings: Default inference uses weighted reflection scores, an adaptive retrieval threshold, segment-level beam search, greedy token decoding, and the top five retrieved documents.The default segment-level beam width is 2.

5 RESULTS AND ANALYSIS

SELF-RAG outperforms strong language-model and retrieval baselines across many tasks, while ablations and analyses show benefits from adaptive retrieval, reflection-guided selection, and test-time customization.

  • Main results: SELF-RAG substantially outperforms supervised fine-tuned LLMs across tasks and surpasses ChatGPT on PubHealth, PopQA, biography generation, and ASQA Rouge and MAUVE.Its 7B and 13B models also outperform CoVe on biography generation.
  • Main results: SELF-RAG achieves the best performance among non-proprietary retrieval models on all tasks and exceeds ChatGPT in ASQA citation precision.Other retrieval baselines show limited gains on PubHealth and ARC-Challenge and generally struggle to improve citation accuracy.
  • Main results: Training with retrieval and reflection yields gains beyond those obtained by using the same instruction-output data with test-time retrieval alone.Llama2-FT7B without retrieval or self-reflection lags behind SELF-RAG.
  • Ablation studies: Ablations show that all evaluated components matter, while always using the top retrieved passage causes large drops on PopQA and ASQA.Removing ISSUP during critique-guided beam search also hurts ASQA, supporting fine-grained multi-criterion generation selection.
  • Inference-time customization: Increasing ISSUP weight improves ASQA citation precision but lowers MAUVE, reflecting a trade-off between support and fluency.The inference procedure lets practitioners adjust these behaviors without additional training.
  • Efficiency and accuracy trade-off: Increasing the retrieval threshold sharply changes retrieval frequency; reduced retrieval harms PopQA more than PubHealth.Larger thresholds produce less retrieval.
  • Training scale: Training-scale experiments compare 5k, 10k, 20k, and 50k subsets with the full 150k training set across PopQA, PubHealth, and ASQA citation precision.The authors report that further expansion may improve performance, although training is limited to 150k instances.
  • Human evaluations: Human annotators find outputs often plausible and supported, while ISREL and ISSUP predictions mostly align with their assessments.The evaluation samples 50 PopQA and biography outputs.

6 CONCLUSION

SELF-RAG trains language models to retrieve, generate, and critique using reflection tokens, enabling retrieval on demand and test-time behavior tailoring. Across six tasks, it outperforms larger LLMs and conventional retrieval-augmented approaches.

  • Conclusion: SELF-RAG enhances language-model quality and factuality through retrieval on demand and self-reflection.The framework trains an LM to retrieve, generate, and critique passages and its own generations.
  • Conclusion: Reflection tokens extend the model vocabulary and enable test-time tailoring of language-model behavior.The same framework supports generation and critique through next-token prediction.
  • Conclusion: Holistic evaluations on six tasks show that SELF-RAG significantly outperforms larger LLMs and conventional retrieval-augmented generation approaches.The conclusion summarizes results across multiple metrics.

ETHICAL CONCERNS

The paper targets factuality problems that can create real-world harms, while acknowledging that SELF-RAG can still produce outputs insufficiently supported by citations.

  • Ethical concerns: Factuality errors in language-model outputs can contribute to misinformation and incorrect or dangerous advice.These harms motivate the paper’s focus on improving factuality.
  • Ethical concerns: SELF-RAG improves performance, factuality, and citation accuracy but can still generate outputs that citations do not fully support.The authors hope explicit self-reflection and fine-grained attribution help users verify errors.

B Experimental Details

The appendix defines reflection-token categories for retrieval, relevance, support, and usefulness, and reports GPT-4 supervision quality for these judgments.

  • Reflection-token definitions: Retrieval-on-demand tokens indicate whether factual grounding is necessary, unnecessary, or should continue using previously retrieved evidence.
  • Reflection-token definitions: ISREL tokens classify retrieved evidence as relevant or irrelevant to the input.
  • Reflection-token definitions: ISSUP tokens assess whether an output is fully supported, partially supported, or unsupported or contradictory relative to evidence.
  • Reflection-token definitions: ISUSE tokens rate whether a response is helpful and informative independently of factuality, using a five-point scale.
  • GPT-4 supervision: 95% agreement was observed for GPT-4 predictions on relevance and retrieval necessity, 90% for support, and 80% for usefulness in a manual sample.

A.2 SELF-RAG TRAINING

SELF-RAG training combines critic-generated reflection labels with retrieved passages to train a generator, while inference scores reflection predictions for controllable decoding.

  • Critic evaluation: The reward model showed high prediction matching with GPT-4-generated feedback.
  • Training pipeline: The generator is trained on data that interleaves input-output text, retrieved passages, and reflection tokens.
  • Critic evaluation: More than 80% accuracy was achieved on most reward-prediction aspects, while usefulness was comparatively lower because adjacent ratings were often confused.
  • Training pipeline: The training pipeline uses a critic to predict retrieval, relevance, support, and usefulness reflection tokens for input-output examples.
  • Inference scoring: Beam-search scores combine normalized probabilities for relevance, support, and five-level usefulness judgments.

B.1 MORE DETAILS OF TRAINING

The appendix specifies hardware, optimization, retrieval, and evaluation settings used to train and test SELF-RAG and its baselines.

  • Training settings: Training used four Nvidia A100 GPUs with 80GB memory, three epochs, batch size 128, and a peak learning rate of 2e-5.
  • Training settings: The 7B model used a 2,048-token maximum length, whereas the 13B model used 1,524 tokens because of memory constraints.
  • Retrieval settings: The default retriever selected the top five Wikipedia documents using Contriever-MS MARCO and 2018 English Wikipedia embeddings.
  • Evaluation settings: Open-domain QA generations were capped at 100 new tokens, while closed-set task outputs were capped at 50 tokens.
  • Evaluation settings: Evaluation instructions varied by task, and Open-domain QA used no explicit instructions.

C.1 ANALYSIS

SELF-RAG relies less often on answers absent from retrieved evidence and flags irrelevant passages before generating potentially unsupported answers.

  • Reliance on evidence: 2% of SELF-RAG’s correct predictions were absent from provided passages, compared with 15% for Alpaca, 18% for Llama2-chat 13B, and 20% for Alpaca 30B.
  • Reliance on evidence: When retrieved passages are irrelevant, SELF-RAG generates an ISREL=Irrelevant signal, whereas instruction-tuned models continue producing plausible answers.

C.2 HUMAN EVALUATION EXAMPLES

The section presents human-evaluation examples for S&P and reflection-token correctness, alongside qualitative SELF-RAG outputs and evaluation instructions. The examples illustrate how reflection tokens support contradiction detection, partial-support judgments, and output verification.

  • Human evaluation: Table 6 presents human evaluations of S&P and the correctness of ISREL and ISSUP reflection tokens.
  • Qualitative examples: Table 7 shows SELF-RAG (13B) examples covering supported claims, contradictions, and partially supported statements.
  • Qualitative examples: Contradictory and Partially Support reflection tokens identify conflicts or omissions between generated statements and cited passages.The examples include a CEO tenure contradiction and a song-listing statement whose names were not explicitly mentioned.
  • Instructions and demonstrations: The appendix documents instructions and demonstrations for collecting Retrieve, ISREL, ISSUP, and ISUSE reflection tokens, as well as training and evaluation examples.
Loading 2310.11511v1…