Source-linked AI summary
Feedback-Assisted Trust Propagation over Document Relation Graphs for Retrieval-Augmented Generation
Zhuoheng Li, Ying Chen
TL;DR
RAG systems face reliability risks because external corpora may contain noisy, outdated, contradictory, or unreliable documents, while prior relation-based methods do not propagate reliability beyond directly compared pairs. TrustPropRAG builds a document-relation graph, anchors multi-hop trust propagation with sparse human feedback, and uses optimized trust scores for retrieval and answer generation; it improves retrieval quality and exact match over baselines and remains robust to noisy feedback.
Problem
External RAG corpora may contain noisy, outdated, contaminated, or conflicting content, while prior document-relation methods do not propagate reliability beyond directly compared documents.
Method
TrustPropRAG constructs a document-relation graph and jointly optimizes pairwise consistency and feedback losses to estimate trust scores for corpus documents.
Results
TrustPropRAG achieves absolute EM gains of 0.03–0.11 over the strongest baseline and improves retrieval quality while remaining robust to noisy feedback.
Takeaways & Limitations
Propagating sparse human feedback through document relations supports selecting more reliable documents and guiding trust-aware answer generation.
Takeaways & Limitations
The evaluation uses binary feedback, synthetic contradictions, and short factual open-domain QA; graded feedback and long-form or multi-hop generation remain future work.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) systems rely on external corpora that may contain outdated, contradictory, noisy, or unreliable documents, introducing reliability risks. Prior work has leveraged document relations to improve the answer reliability of RAG. To propagate reliability signals beyond directly compared document pairs, we propose TrustPropRAG, which structures document relations as a graph and estimates document reliability through multi-hop propagation across the graph. TrustPropRAG anchors this propagation with a limited set of human feedback on document reliability, extending these costly-to-collect feedback-based reliability signals across the whole corpus. Specifically, based on the constructed document relation graph, TrustPropRAG estimates a trust score for each document by formulating and solving an optimization problem that jointly captures pairwise document relations and user feedback. These scores are then used to improve the selection of reliable documents and support trust-aware answer generation. Evaluation results show that TrustPropRAG improves both retrieval quality and exact match over baselines, and remains robust under sparse and noisy feedback.
1 Introduction
TrustPropRAG addresses reliability risks in external RAG corpora by propagating sparse human feedback through document-relation graphs, using resulting trust scores to improve retrieval and answer generation.
- Motivation: External RAG corpora can contain noisy, outdated, contaminated, or conflicting content, creating reliability risks during answer generation.RAG supplements incomplete or outdated parametric knowledge with retrieved external information, but inherits risks from that corpus.
- Prior Work: Prior reliability methods use document relationships for answer aggregation or consistency-based document selection, with relations inferred through NLI models or source metadata.These approaches primarily compare or combine documents to improve retrieval or generation reliability.
- Approach: TrustPropRAG structures document relations as a graph to propagate reliability estimates across multi-hop connections.Unlike graph-enhanced RAG methods focused on knowledge organization or retrieval indexing, this framework uses graph structure to estimate document reliability.
- Approach: Because document relations reveal consistency or conflict but not reliability itself, TrustPropRAG anchors propagation with explicit or implicit human feedback.Traceback methods can attribute ratings, clicks, or dwell-time signals to documents contributing to generated answers.
- Approach: TrustPropRAG estimates document trust scores by jointly modeling pairwise relations and user feedback, then uses them for trust-aware retrieval rescoring.The framework constructs a graph whose nodes are documents and whose edges encode relations such as shared sources, mutual support, or contradiction.
- Evaluation: The framework is evaluated across three QA benchmarks, three retrievers, and six LLMs, including robustness tests with sparse and noisy feedback.Its stated contributions include trust-score optimization, theoretical analysis of feedback propagation and convergence, and broad empirical evaluation.
2 Related Work
Prior RAG reliability methods use document relationships within retrieved subsets, whereas TrustPropRAG represents those relationships as a graph to propagate reliability beyond directly compared documents.
- Relationship-Based Reliability: Prior methods improve reliability by aggregating independently generated document answers or filtering documents using agreement and contradiction relationships.Examples include majority agreement, clustering-based contamination filtering, and mutually consistent document selection.
- Limitations of Prior Work: These approaches evaluate relationships within a document subset and do not propagate reliability beyond directly compared documents.Their reliability reasoning is therefore limited to one-hop comparisons among selected documents.
- TrustPropRAG: TrustPropRAG structures document relations as a graph to capture multi-hop connections and estimate reliability across the corpus without model fine-tuning.The graph enables reliability signals to extend beyond the documents directly compared by prior methods.
3 Methodology
TrustPropRAG estimates document trust from a relation graph anchored by human feedback, then uses those scores with query relevance to select retrieval context. Its optimization combines pairwise consistency and feedback objectives, while trust signals propagate across graph paths.
- Document relation graph: TrustPropRAG constructs a document relation graph whose nodes are documents and whose edges encode supportive, contradictory, or other relations.Relations can come from source metadata or NLI predictions.
- Trust score optimization: The method assigns every document a trust score by jointly optimizing pairwise consistency and feedback losses.Positive relations encourage similar scores, contradictory relations encourage scores summing toward one, and feedback pushes marked documents toward high or low targets.
- Trust score optimization: Trust score optimization is a quadratic program solved with projected gradient descent over sparse graph-connected document pairs.The sparse graph avoids computing pairwise losses for all possible document pairs.
- Trust propagation: Human feedback propagates through document relations: each document receives cumulative, distance-weighted influence from all feedback documents.The influence depends on shortest-path distance, so well-positioned sparse feedback can reach other documents through short paths.
- Trust-aware retrieval: At query time, TrustPropRAG combines estimated trust with retriever similarity to rank documents for answer-generation context.Graph construction and trust optimization occur offline, while rescoring and prompting occur per query.
4 Evaluation
TrustPropRAG is evaluated across diverse QA datasets, retrievers, LLMs, feedback conditions, and ablations. It improves exact match and retrieval factuality, remains robust to sparse and noisy feedback, and adds limited query-time overhead after offline processing.
- Evaluation setup: Evaluation covers MS MARCO, Natural Questions, and TriviaQA using BM25, Contriever, and MiniLM retrievers with k=5.The study also evaluates proprietary and open-source LLMs, including GPT-4o-mini, GPT-5, Gemini models, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct.
- Overall results: TrustPropRAG achieves the best EM across all datasets and retrievers, with absolute gains of 0.03–0.11 over the strongest baseline.EM is averaged over GPT-4o-mini, Gemini 2.5 Flash, and GPT-5.
- Overall results: 77–96% FP@5 for TrustPropRAG exceeds the 38–57% achieved by baselines across all three retrievers.FP@5 measures the fraction of factual documents among the top-k retrieved documents.
- Ablations: Trust propagation outperforms directly using feedback: TrustPropRAG reaches EM 0.49 and FP@5 92%, versus 0.36 and 62% for trivial feedback.The NLI-only variant still achieves EM 0.44 and FP@5 79%, despite removing source-group edges.
- Feedback robustness: At 60% feedback accuracy, TrustPropRAG still improves EM over no-feedback settings across all feedback ratios.Performance improves sharply when feedback covers 10% of documents, then improves more gradually as the ratio increases.
- Trust coefficient: EM is best at α = 0.6 and remains stable from 0.5 to 0.8, while overly large α slightly reduces EM by favoring less relevant high-trust documents.Trust-aware rescoring improves both EM and FP@5 as α increases from zero.
- Different LLMs: TrustPropRAG improves average EM by 0.17–0.23 over VanillaRAG across proprietary and smaller open-source LLMs.The average gain is 0.23 with GPT-5 and Gemini 2.5 Pro.
- Computational cost: Graph construction costs 578.3–1077.1 seconds offline, whereas trust optimization takes 0.3–0.4 seconds and adds negligible propagation overhead.Query-time retrieval takes 4.4–4.8 seconds for both TrustPropRAG and baselines; trust-aware rescoring adds limited overhead.
5 Conclusions
TrustPropRAG improves RAG reliability by propagating sparse human feedback across a document relation graph. It estimates document trust through joint pairwise-consistency and feedback optimization, then uses those scores for reliable retrieval and answer generation.
- TrustPropRAG propagates sparse human feedback over a document relation graph to estimate document trust.
- The framework optimizes an objective combining pairwise consistency loss and feedback loss.
- Resulting trust scores select more reliable documents and guide trust-aware answer generation.
- TrustPropRAG achieves absolute EM gains of 0.03–0.11 over the strongest baseline and remains robust to noisy feedback.
Limitations
TrustPropRAG’s evaluation and formulation have defined scope boundaries. The method uses binary feedback and synthetic contradictions, and focuses on open-domain QA with short factual answers.
- TrustPropRAG represents human feedback with binary reliable or unreliable labels.Graded feedback is left for future work.
- The evaluation injects unreliable content as synthetically generated contradictions rather than naturally occurring conflicts.
- The study focuses on open-domain QA with short factual answers.Applying trust propagation to long-form or multi-hop generation is left for future work.
Ethical Considerations
The work addresses unreliable content in retrieval-augmented generation using publicly released datasets and models licensed for research use. Its experiments use synthetically generated contradictions.
- The study addresses unreliable content in retrieval-augmented generation.
- Experiments use synthetically generated contradictions to represent unreliable content.
- All datasets and models used are publicly released and licensed for research use, consistent with their intended purposes.
A Implementation Details
The implementation evaluates TrustPropRAG with proprietary and open-source language models, while using fixed hyperparameters across datasets, retrievers, and LLMs. The setup includes trust-score initialization and feedback-loss settings.
- Evaluated LLMs include open-source Llama-3.1-8B-Instruct and Mistral-7B-Instruct, plus proprietary GPT-4o-mini, GPT-5, Gemini 2.5 Flash, and Gemini 2.5 Pro.
- The NLI model has approximately 44M parameters, while Contriever and all-MiniLM-L6-v2 have approximately 110M and 22M parameters.
- Table 7 examines how the number of retrieved documents k affects EM and FP@k on MS MARCO with MiniLM, Gemini 2.5 Flash, and FB=30%.
- The implementation uses the same hyperparameters across all datasets, retrievers, and LLMs.
- Trust scores are initialized with τmin = 0.3 and τmax = 0.7, with λf=1.0 and feedback values τ+ = 0.9 and τ−=0.1.
B Number of Retrieved Documents k
The number of retrieved documents trades off context coverage against the risk of including unreliable evidence. Trust-aware rescoring performs best with a selective context, while larger contexts remain manageable for answer generation.
- Smaller k provides selective context but may omit relevant evidence, whereas larger k broadens coverage while admitting more unreliable documents.
- 95% to 73%: FP@k falls as k increases from 1 to 20.Lower-trust documents increasingly enter the selected top-k set.
- EM stays within 0.41–0.46 across k values and peaks at k=5 and k=10.At k=1, the context may provide insufficient evidence.
- Trust-aware rescoring places reliable evidence first, helping the LLM discount lower-trust documents when larger contexts are supplied.
- The experiments adopt k=5, matching prior work because EM peaks at k=5 and k=10.
C Proofs
The proofs analyze trust propagation through a quadratic optimization problem and establish conditions for positive definiteness, perturbation decay, and convergence. The system also uses projected gradient descent and trust-aware prompting in the RAG pipeline.
- Trust score optimization is quadratic, with first-order optimality expressed as the linear system Aτ* = b.A is the Hessian of the objective.
- A is positive definite when every connected component of G contains at least one document with feedback.This anchoring assumption supports the proof of the lemmas.
- Feedback perturbations leave A unchanged and modify only the right-hand side b at feedback-document entries.The resulting trust-score changes are expressed through A^-1.
- The sparsity pattern of A follows the document-graph adjacency, enabling inverse-entry decay with graph distance under the stated matrix conditions.
- Mislabeled-edge influence on optimized trust scores decreases exponentially with shortest-path distance from the erroneous edge.The effect of an edge error therefore diminishes rapidly for more distant documents.
- The trust-aware prompt annotates documents with trust scores and labels, instructing the LLM to prefer HIGH-trust and discount LOW-trust documents.Labels are high for τi ≥0.7, low for τi ≤0.3, and medium otherwise.
F Quality of NLI-derived Edges
NLI-derived document relations are imperfect: they have moderate precision and low recall because most document pairs are classified as neutral. Despite this sparse, noisy graph, the method remains effective in the reported ablation.
- NLI-derived edges have moderate precision and low recall because the model predicts most document pairs as neutral.
- Low recall omits many relations, producing a sparse graph that reflects practical settings where only a limited subset of relations is identifiable.
- Imperfect precision means the graph contains some incorrectly labeled edges.
- The NLI-only ablation still outperforms the reported comparison baseline despite these sparse and imperfect relations.The supplied passage states this outcome without giving the comparison values.
G Performance on QACC
TrustPropRAG is evaluated on QACC, a naturally conflicting corpus built from real web contexts. It achieves the best reported exact match, indicating effectiveness beyond synthetically injected contradictions.
- QACC pairs unambiguous questions with real web contexts retrieved through Google Search.
- 0.59 EM: TrustPropRAG achieves the best exact match on the naturally conflicting QACC corpus.
- TrustPropRAG exceeds ReliabilityRAG at 0.56 EM and improves over VanillaRAG by 0.11.
- The results indicate that TrustPropRAG's effectiveness extends beyond synthetically injected contradictions to naturally occurring web conflicts.