PaperlayerView exact revision on arXiv

Source-linked AI summary

Trustworthiness in Retrieval-Augmented Generation Systems: A Survey

Yujia Zhou, Wenbo Zhang, Jingying Shao, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Jason Chen Zhang, Zhicheng Dou, Philip S. Yu, Jiaxin Mao

arXiv:2409.10102v2cs.IRcs.AIcs.CL

TL;DR

LLMs can hallucinate, and RAG’s use of external information does not by itself settle questions of trustworthiness. The paper defines six dimensions, reviews related work, and evaluates 19 LLMs with TRC Bench. Across most dimensions, proprietary models generally perform more strongly than open-source models.

  • Problem

    LLMs can produce plausible but incorrect information, motivating RAG’s use of external information to improve accuracy and credibility.

    Source: §1 · p. 3

  • Method

    The paper defines six trustworthiness dimensions, reviews existing work, and introduces TRC Bench to evaluate 19 proprietary and open-source LLMs.

    Source: §1 · pp. 3–4

  • Results

    Proprietary LLMs generally show stronger trustworthiness performance than open-source models across most dimensions, especially factuality, accountability, transparency, and privacy.

    Source: §4.2.2 · p. 28

  • Takeaways & Limitations

    The survey offers a six-dimension framework, literature review, and benchmark evaluation as actionable insights and guidelines for future RAG system development.

    Source: §1 · p. 4

    Some instruction-tuned open-source models still underperform proprietary models, especially in factuality and accountability, and alignment does not fully solve trustworthiness problems.

    Source: §4.2.2 · p. 29

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has quickly grown into a pivotal paradigm in the development of Large Language Models (LLMs). Although existing research mainly emphasizes accuracy and efficiency, the trustworthiness of RAG systems remains insufficiently explored. RAG can improve LLM reliability by grounding responses in external and up-to-date knowledge, reducing hallucinations. However, unreliable retrieval or improper knowledge utilization may still lead to undesirable outputs. To address these concerns, we propose a unified framework, Trust-RAG Compass, that assesses the trustworthiness of RAG systems across six key dimensions: factuality, robustness, fairness, transparency, accountability, and privacy. Within this framework, we provide a thorough review of the existing literature along each dimension. Furthermore, we introduce an evaluation benchmark, TRC Bench (\underline{T}rust-\underline{R}AG \underline{C}ompass \underline{Bench}mark), regarding the six dimensions and conduct comprehensive evaluations for a variety of proprietary and open-source models. Our results shed light on the performance gaps between different types of LLMs across varying dimensions of trustworthiness. Finally, we identify key challenges and promising directions for future research based on our findings. Through this work, we aim to provide a structured foundation for subsequent investigations and practical guidance for developing trustworthy RAG systems in real-world scenarios.

1 Introduction

RAG can ground LLM responses in external information, but trustworthiness remains underexplored relative to retrieval and generation optimization. The paper defines a six-dimension framework, reviews existing work, and evaluates 19 LLMs with TRC Bench.

1 Introduction

  • RAG uses external information to improve accuracy and credibility, but research still focuses mainly on retriever and generator optimization rather than trustworthiness.The paper identifies defining comprehensive trustworthiness dimensions and developing practical evaluation methodologies as two challenges.Source: §1 · p. 3
  • The framework defines factuality, robustness, fairness, transparency, accountability, and privacy, and reviews approaches across these dimensions.It describes factuality as accurate information verified against reliable sources, robustness as reliability against errors and threats, and the remaining dimensions as addressing bias, clarity, traceability, and privacy.Source: §1 · p. 3
  • TRC Bench evaluates 19 proprietary and open-source LLMs with different sizes and training strategies across six trustworthiness dimensions.The benchmark is intended to provide insights into model trustworthiness in real-world RAG applications.Source: §1 · p. 4
  • The survey’s three contributions are a six-dimension framework, a literature review identifying gaps and promising approaches, and a 19-model benchmark evaluation.The authors also report actionable insights and guidelines for future RAG system development.Source: §1 · p. 4

2 Background and Preliminaries

The background traces RAG from retrieval-then-read systems to more adaptive architectures, then frames trustworthiness risks in LLMs and RAG. The paper analyzes those risks across six dimensions.

2.1 Retrieval-Augmented Generation System

  • RAG has progressed from Naive retrieval-then-read systems through Advanced pre- and post-retrieval components to Modular and Agentic systems with adaptive, multi-step orchestration.Modular pipelines may be sequential, conditional, branching, or loop-based; agentic approaches add planning, reflection, tool use, and coordination.Source: §2.1 · pp. 4–5

2.2 Trustworthiness in Large Language Models

  • LLM trustworthiness risks include hallucination, discrimination, and privacy violations, associated with problematic training data and algorithmic behavior.The paper notes that diverse training corpora can contain harmful content and bias, while Transformer models may capture superficial correlations and probabilistic generation can produce nonfactual text.Source: §2.2 · p. 5
  • RAG can reintroduce risks such as data leakage and unfairness when retrieved content contains sensitive personal information.The model may inadvertently disclose that information in its output.Source: §2.2 · p. 5
  • The paper analyzes RAG trustworthiness across factuality, robustness, fairness, transparency, accountability, and privacy.Figure 2 depicts the integration of the six evaluation dimensions within the complete RAG framework.Source: §2.2 · p. 5, Figure 2 · p. 6

3 Trustworthy RAG System

The paper frames a complete RAG system as a three-stage process, with trustworthiness challenges at each stage. It organizes its analysis around six dimensions and uses a study timeline to categorize the literature.

3 Trustworthy RAG System

  • A complete RAG system has three stages: injecting external knowledge into the generator, generating answers, and evaluating those answers.Trustworthiness issues can arise during each stage.Source: §3 · p. 6
  • Knowledge injection risks noisy or private information, answer generation can introduce biased reasoning or compromise RLHF alignment, and answer evaluation can miss factual errors or grounding.Source: §3 · p. 6
  • The paper examines robustness, fairness, factuality, privacy, transparency, and accountability through definitions and literature review, using a study timeline to identify trends.Source: §3 · p. 6

3.1 Factuality

The section treats factuality as accurate, coherent, temporally appropriate information, then extends the concern to how RAG systems reconcile internal knowledge with retrieved evidence. It reviews methods for integration, dynamic retrieval, and adaptive search.

3.1.1 General Definition for LLMs.

  • For LLMs, factuality concerns accurate information, truthfulness, logical consistency, temporal awareness, and adherence to instructions.Source: §3.1.1 · pp. 6–7

3.1.2 Factuality in RAG Systems.

  • In RAG, factuality requires synthesizing internal and retrieved knowledge, creating challenges beyond factuality in vanilla generation.Source: §3.1.2 · p. 7
  • RAG factuality is challenged by conflicts between internal and external knowledge, noisy retrieved documents, and reasoning over long, structurally complex contexts.Multi-hop questions require accurate inference across multiple pieces of information.Source: §3.1.2 · p. 7

3.1.3 Representative Studies.

  • The review categorizes studies by trustworthiness dimension, method type, and object; Table 1 compares representative factuality methods using those categories.Figure 3 presents a timeline of representative studies across six trustworthiness dimensions through March 2026.Source: §3.1.3 · p. 7, Table 1 · p. 8, Figure 3 · p. 10
  • Integration approaches include SAIL’s search-augmented instruction tuning, Replug’s per-document generation paths and token-distribution merging, and methods for reliability feedback or extracting internal knowledge.FoRAG uses an outline-enhanced generator and fine-grained RLHF; FRAMES evaluates factuality, retrieval, and reasoning across 824 primarily high-difficulty multi-hop test instances.Source: §3.1.3 · pp. 7–8
  • Dynamic retrieval methods include query decomposition, confidence-triggered retrieval, chain-of-thought-guided search, and self-reflective document selection.Self-RAG reports factual-accuracy and response-quality improvements across tasks; ReaRAG iteratively retrieves to refine reasoning trajectories.Source: §3.1.3 · p. 9
  • Adaptive search agents coordinate retrieval decisions, internal memory, and external evidence as a multi-step reasoning-and-search process.ActiveRAG uses knowledge assimilation and thought accommodation agents, while MMOA-RAG jointly optimizes pipeline agents under a unified reward.Source: §3.1.3 · p. 9
  • SCMRAG uses a dynamic knowledge graph for corrective retrieval, MADAM-RAG uses multi-agent debate for conflicting evidence, and SAFE uses agentic extraction and verification for COVID-19 fact-checking.SAFE reports improved consistency, usefulness, clearness, and authenticity over baseline LLMs.Source: §3.1.3 · p. 10

3.2 Robustness

The survey frames RAG robustness around reliable knowledge use despite variation in retrieved information, and reviews attacks and defenses across the retrieval–generation pipeline.

3.2.1 General Definition for LLMs.

  • LLM robustness covers stable performance across diverse inputs and environments, including input diversity, noise tolerance, adversarial resistance, and distribution shifts.Source: §3.2.1 · p. 10

3.2.2 Robustness in RAG Systems.

  • RAG robustness concerns consistently extracting and using relevant knowledge across retrieval inputs that vary in noise, detail, order, and accuracy.The review organizes representative work around attack mechanisms and defense strategies.Source: §3.2.2 · p. 11, §3.2.3 · p. 11

3.2.3 Representative Studies.

  • RAG attacks span retrieval-stage, backdoor-based, prompt- and interaction-level, and structural mechanisms that intervene at different points in the pipeline.The taxonomy describes a shift from unconditional data poisoning toward stealthier, conditional manipulation strategies.Source: §3.2.3 · p. 11
  • Corpus poisoning can steer retrieval toward misleading documents, while trigger-based backdoors preserve normal behavior on benign queries and activate under specific triggers.Examples include PoisonedRAG [197] and BadRAG [170].Source: §3.2.3 · pp. 11–13
  • Structural attacks exploit retrieval order, textual perturbations, and retrieval constraints, including typo-induced degradation and blocker documents occupying top-k slots.GARAG demonstrates perturbation sensitivity; Shafran et al. [127] describe jamming attacks.Source: §3.2.3 · p. 13
  • Defenses include evidence filtering, robust aggregation, robust training, and adaptive retrieval, with agent-based approaches making retrieval more iterative and self-correcting.Filtering targets noisy or malicious documents, while aggregation strategies address conflicting or adversarial evidence.Source: §3.2.3 · pp. 13–14

3.3 Fairness

The survey treats fairness in RAG as an underdeveloped concern shaped by both external knowledge and system behavior, and reviews work from problem identification to adaptive mitigation.

3.3.1 General Definition for LLMs.

  • LLM fairness means avoiding propagated bias and treating individuals and groups equitably; mitigation can occur before, during, or after training.The survey distinguishes data fairness, algorithm fairness, bias detection, and bias mitigation.Source: §3.3.1 · pp. 15–16

3.3.2 Fairness in RAG Systems.

  • Unlike vanilla generation, RAG can inherit bias from external knowledge bases as well as training data, making knowledge-source imbalance an additional concern.Imbalanced sources can over-represent some viewpoints and marginalize others, including non-Western perspectives in a Western-skewed base.Source: §3.3.2 · p. 16
  • Unreliable or biased sources can perpetuate inaccuracies and stereotypes, while biased retrieval rankings can create filter bubbles that reinforce prior user preferences.Source: §3.3.2 · p. 16
  • During integration, models may favor evidence aligned with parametric biases, and relevance-based integration without fairness checks can amplify small biases in retrieved documents.Source: §3.3.2 · pp. 16–17

3.3.3 Representative Studies.

  • Existing fairness approaches often treat fairness statically, whereas newer agent-based frameworks integrate bias detection, source selection, and evidence filtering into retrieval.The survey describes this as a shift toward treating fairness as part of retrieval and reasoning rather than post hoc correction.Source: §3.3.3 · pp. 17–18

3.3 Fairness

  • Fairness in RAG is a multi-stage challenge arising from interactions among external knowledge, retrieval algorithms, and generation mechanisms.The survey describes fairness research as early-stage and categorizes studies by trustworthiness dimension, method type, and object.Source: §3.3 · p. 15, §3.3.3 · pp. 17–18

3.4 Transparency

The survey organizes RAG transparency around understanding retrieval and evidence integration, then reviews approaches for reasoning regulation, explanations, and evaluation.

3.4.1 General Definition for LLMs.

  • LLM transparency research seeks to explain how models process information, make decisions, and generate outputs through data, model, algorithm, and explanation transparency.Source: §3.4.1 · p. 18

3.4.2 Transparency in RAG Systems.

  • RAG transparency includes examining how retrieval selects and ranks documents and how retrieved information is integrated into generated answers.Tracing generated content to particular retrieved documents can clarify information lineage.Source: §3.4.2 · pp. 18–19

3.4.3 Representative Studies.

  • MetaRAG uses monitoring, evaluation, and planning to let models introspect and adjust multi-hop reasoning, including diagnosing insufficient knowledge, conflicting information, and reasoning errors.Source: §3.4.3 · p. 19
  • RAG-Ex provides perturbation-based explanations for open-source and proprietary LLMs, while RAGBench and TRACe offer a large-scale benchmark with context-utilization and answer-completeness metrics.RAGBench contains 100k examples across industry-specific domains.Source: §3.4.3 · p. 19

3.5 Accountability

The survey extends accountability from output attribution to oversight of the full RAG pipeline, reviewing methods for provenance, verification, reasoning traces, and responsibility.

3.5.1 General Definition for LLMs.

  • LLM accountability concerns tracing and justifying outputs and assigning responsibility, supported by traceability, verifiability, and responsibility assignment.Documentation, audit trails, and model versioning support post hoc analysis and error diagnosis.Source: §3.5.1 · pp. 19–20

3.5.2 Accountability in RAG Systems.

  • RAG accountability covers retrieval, generation, and their interface, requiring validation of sources and tracking which retrieved evidence influenced generated content.The survey characterizes it through source attribution, evidence verification, reasoning traceability, and responsibility assignment.Source: §3.5.2 · p. 20

3.5.3 Representative Studies.

  • Research progresses from output-level citations toward evidence verification, including methods that assess claim–evidence consistency and post-generation factual validation.AGREE and VTG [140] use NLI models; RARR [44] and CEG [86] perform post-generation editing and validation.Source: §3.5.3 · pp. 20–21
  • Reasoning-trace methods expose intermediate evidence use, while agent-based frameworks coordinate attribution, verification, and trust modeling as process-level accountability.The survey describes a shift toward explicitly modeling and continuously verifying evidence provenance and reasoning trajectories.Source: §3.5.3 · pp. 21–22

3.6 Privacy

The section frames privacy in RAG as a multi-stage challenge spanning sensitive data, retrieval, and generation. It reviews extraction, inference, and retrieval leakage risks alongside defenses, including approaches that embed privacy protection across the pipeline.

3.6.1 General Definition for LLMs.

  • LLM privacy risks include inadvertent disclosure of personal data in responses and prompt-based extraction or inference of information learned during training.RAG adds risks involving sensitive retrieval databases, including access to private medical records through crafted queries.Source: §3.6.1 · p. 22, §3.6.2 · p. 22

3.6.2 Privacy in RAG Systems.

  • RAG privacy risks include direct exposure of database contents, inference about whether data is present, and retrieval that selects and exposes private documents.Black-box attackers have extracted database information by crafting prompts [117].Source: §3.6.2 · p. 22

3.6.3 Representative Studies.

  • Studies distinguish direct extraction attacks [117] [110], membership inference [4], and retrieval-induced leakage [178] [62].The cited work reports that kNN-LMs are more susceptible to datastore leakage than purely parametric models [62].Source: §3.6.3 · p. 23
  • Defenses include privacy filtering and query-specific thresholds in MURAG methods, which support hundreds of queries under practical privacy budgets while maintaining meaningful task utility.MURAG’s accumulated privacy loss depends primarily on each document’s retrieval frequency rather than the total number of queries.Source: §3.6.3 · p. 24
  • Other defenses use secret sharing for retrieval [100] or encrypt text and embeddings throughout retrieval [188].The p2RAG approach also uses an interactive bisection method to determine the top-k document set; Privacy-Aware RAG reports formal security analysis.Source: §3.6.3 · p. 24
  • Agent-based approaches use synthetic corpora, privacy agents, adaptive masking, and decentralized coordination to protect data across retrieval and collaboration [177, 130, 15, 172].The section characterizes this direction as a shift from static access control toward dynamic, process-level protection.Source: §3.6.3 · p. 24

4 Evaluation

The evaluation introduces TRC Bench, a six-dimension benchmark with dimension-specific methods for comparing LLM trustworthiness in RAG. Its evaluations test factuality, robustness, fairness, transparency, accountability, and privacy using task-specific datasets and measures.

4.1 Benchmarking and Evaluation Methods

  • TRC Bench covers six trustworthiness dimensions and provides dimension-specific evaluation methods for comparing LLMs in RAG settings.The benchmark datasets and evaluation code are publicly available at the TrustworthyRAG Repository.Source: §4.1 · p. 24

4.1.1 Factuality Evaluation.

  • Factuality evaluation tests whether models avoid incorrect answers when retrieved documents contain misleading but relevant information.The evaluation uses 50 RGB samples and measures the proportion of answers that omit the dataset’s fake answers.Source: §4.1.1 · pp. 24–25

4.1.2 Robustness Evaluation.

  • Robustness evaluation varies the ratio of irrelevant to relevant content in sets of 3 or 10 references, each containing 2 golden references.It uses 50 HotpotQA questions and reports F1 scores and degradation across the two noise settings.Source: §4.1.2 · p. 25

4.1.3 Fairness Evaluation.

  • Fairness evaluation adds biased information to retrieved documents and tests whether models support biased gender statements in reconstructed yes-or-no questions.The questions come from 50 samples drawn from the gender dimension of CrowS-Pair.Source: §4.1.3 · p. 25

4.1.4 Transparency Evaluation.

  • Transparency evaluation uses key-facts and an NLI entailment oracle to measure reasoning-step recall, statement precision, and fact density.GPT-4 assists in constructing key-facts; the evaluation uses TRUE [55] as its entailment oracle. High recall with low fact density indicates verbose, redundant reasoning.Source: §4.1.4 · p. 26

4.1.5 Accountability Evaluation.

  • Accountability is evaluated through citation F1, combining citation precision—the accuracy of included citations—and recall—the completeness of relevant citations.The setup supplies ten retrieved documents and samples 50 HotpotQA questions.Source: §4.1.5 · p. 27

4.1.6 Privacy Evaluation.

  • Privacy evaluation measures refusal rates on questions about email addresses using the Enron corpus as the retrieval database.The setup uses all approximately 500,000 emails, 50 sampled questions, and BM25 top-3 retrieval, without explicitly instructing the generator to protect privacy.Source: §4.1.6 · p. 27

4.2 Evaluation Results and Analysis

Across 19 open-source and proprietary LLMs, the evaluation finds stronger proprietary-model performance on several trustworthiness dimensions, alongside persistent weaknesses in fairness and privacy. Results also show differences associated with instruction tuning and model scale, while alignment does not eliminate open-source limitations in complex RAG settings.

4.2. Evaluation Results and Analysis

  • The evaluation tests 19 open-source and proprietary models under the same datasets, retrieval corpora, and prompting settings across six trustworthiness dimensions.The models span multiple families, scales, and alignment strategies.Source: §4.2 · p. 27, §4.2.1 · pp. 27–28

4.2.2 Overall Analysis.

  • Proprietary models generally outperform open-source models in factuality, accountability, transparency, and privacy, though fairness remains a weakness across both groups.Gemini-3.1-pro has the best factuality score; Claude models lead transparency, while GPT and Claude models score highly in accountability.Source: Table 9 · p. 28, §4.2.2 · pp. 28–29
  • Instruction-tuned models generally outperform their base counterparts, especially in robustness, accountability, and transparency.Figure 4 shows instruct variants are more robust across reference settings; Figure 5 reports higher accountability precision, recall, and F1 scores for instruct models.Source: §4.2.2 · p. 29, Figure 4 · p. 30
  • The Ministral series particularly illustrates improvements in accountability and transparency from base to instruct variants.The authors suggest that without explicit alignment, models may not effectively ground answers in retrieved evidence.Source: §4.2.2 · p. 29
  • Alignment does not resolve all open-source shortcomings: some instruction-tuned models still underperform proprietary models in factuality and accountability in complex RAG scenarios.These scenarios require distinguishing reliable evidence, ignoring noisy references, and producing concise yet complete answers.Source: §4.2.2 · p. 29
  • Larger models can improve some dimensions within a model family, but size alone does not ensure consistent trustworthiness gains across dimensions.Qwen3.5-27B is stronger than Qwen3.5-4B and Qwen3.5-9B in robustness, transparency, and accountability; larger Ministral instruct models also tend to improve robustness.Source: §4.2.2 · p. 29
  • Fairness and privacy are greater challenges than robustness and accountability; open-source privacy is weak, and aligned variants can score 0.Privacy also varies among proprietary models: Gemini models and GPT-5.4-mini perform worse than GPT-5.4 and GPT-5.4-pro.Source: Table 9 · p. 28, §4.2.2 · pp. 29–30
  • Open-source models are more sensitive to changes in retrieved references, while transparency involves a completeness–efficiency trade-off across model groups.Some models have high recall but lower precision or fact density; the authors recommend concise, relevant, evidence-grounded reasoning.Source: §4.2.2 · p. 30, Figure 4 · p. 30

5 Main Challenges and Future Works

The paper describes trustworthiness challenges that intensify in agentic RAG, where retrieval, reasoning, and action are coupled across iterative decision-making. It proposes agent-centric research directions spanning data, retrieval, tools, memory, evaluation, and governance.

5.1 Main Challenges

  • Agentic RAG can propagate factual errors when static model knowledge conflicts with retrieved evidence across iterative reasoning, especially as long contexts and memory accumulate.The paper calls for adaptive retrieval and context management to maintain factual consistency across extended reasoning trajectories.Source: §5.1 · pp. 30–31
  • Multi-step tool use adds robustness risks from error amplification, misleading outputs, long-horizon dependencies, and tool reliability or interface design.The paper argues for end-to-end modeling of the decision loop rather than isolated stage improvements.Source: §5.1 · p. 31
  • Fairness risks arise as biases accumulate across retrieval, reasoning, and aggregation of multi-source or multimodal data.Proposed interventions include fairness-aware retrieval, source balancing, and bias monitoring during iterative reasoning.Source: §5.1 · pp. 31–32
  • Agentic systems complicate transparency because users need to understand tool choices, intermediate reasoning, and how evidence changes over time.The paper says current explanation techniques do not capture this process-level transparency adequately.Source: §5.1 · p. 32
  • Accountability requires tracking evidence provenance and intermediate reasoning, while integrated tools and persistent memory amplify privacy risks.Privacy exposure can arise through private-corpus retrieval, long-term memory, or unintended inference; proposed protections span retrieval filtering, memory control, and privacy-aware reasoning.Source: §5.1 · pp. 32–33

5.2 Future Works

  • Future data curation should capture retrieval and reasoning trajectories, tool-use patterns, and human-annotated decisions, while retrieval policies adapt to evidence quality.The paper also calls for explicit ethical considerations in high-stakes domains such as medicine and biology.Source: §5.2 · p. 33
  • Future work should strengthen agent robustness through adversarial training and tool-output uncertainty modeling, while managing memory compression, contamination, and drift.The paper frames memory management as important for both performance and privacy.Source: §5.2 · p. 33
  • The paper calls for process-aware benchmarks and integrated control frameworks that assess reasoning steps, tool use, fairness, privacy, and accountability.It recommends holistic agent-centric RAG frameworks jointly optimizing factuality, robustness, fairness, accountability, and privacy.Source: §5.2 · p. 33

6 Conclusion

The paper defines trustworthiness for LLMs in RAG settings, reviews related work, establishes benchmarks and evaluation methods, and analyzes 19 models. It identifies shortcomings and challenges, then outlines future research directions for trustworthiness in real-world and high-stakes applications.

6 Conclusion

  • The paper defines RAG trustworthiness across six dimensions and reviews related work while establishing benchmarks, evaluation methods, and an empirical analysis of 19 models.The conclusion names actuality, transparency, accountability, privacy, fairness, and robustness.Source: §6 · p. 33
  • The evaluation uncovers significant model shortcomings and summarizes the challenges they face.The paper also outlines promising avenues for future research.Source: §6 · pp. 33–34
  • The authors frame trustworthiness as increasingly critical for responsible and ethical deployment in real-world applications, particularly healthcare, scientific research, and decision support.They state that addressing these challenges will enhance utility and support responsible deployment.Source: §6 · p. 34

Related papers

Loading 2409.10102v2…