Source-linked AI summary
A Decentralized Retrieval Augmented Generation System with Source Reliabilities Secured on Blockchain
Yining Lu, Wenyi Tang, Max Johnson, Taeho Jung, Meng Jiang
TL;DR
Centralized RAG can raise data-management costs and privacy concerns, while decentralized sources vary in reliability. dRAG dynamically scores sources and manages scores through blockchain smart contracts. It outperforms centralized systems in unreliable data environments and can approach performance with fully reliable sources.
Problem
Centralized RAG raises data-management cost and privacy concerns, while decentralized sources can vary in reliability and affect retrieval and response quality.
Source: §1 · p. 1Method
dRAG scores sources according to the quality of responses they help generate, prioritizes high-quality sources, and manages scores and logs through blockchain smart contracts.
Source: §1 · pp. 1–2Results
dRAG consistently outperforms centralized systems in unreliable data environments and achieves performance comparable to fully reliable environments.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
Data owners retain control over what information to share and which retrieval policies to implement while language models retrieve from their sources.
Source: §1 · p. 1The number of queries needed to reach dRAG’s performance upper bound and the factors affecting reliability-score convergence remain unclear.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Existing retrieval-augmented generation (RAG) systems typically use a centralized architecture, causing a high cost of data collection, integration, and management, as well as privacy concerns. There is a great need for a decentralized RAG system that enables foundation models to utilize information directly from data owners who maintain full control over their sources. However, decentralization brings a challenge: the numerous independent data sources vary significantly in reliability, which can diminish retrieval accuracy and response quality. To address this, our decentralized RAG system has a novel reliability scoring mechanism that dynamically evaluates each source based on the quality of responses it contributes to generate and prioritizes high-quality sources during retrieval. To ensure transparency and trust, the scoring process is securely managed through blockchain-based smart contracts, creating verifiable and tamper-proof reliability records without relying on a central authority. We evaluate our decentralized system with two Llama models (3B and 8B) in two simulated environments where six data sources have different levels of reliability. Our system achieves a +10.7\% performance improvement over its centralized counterpart in the real world-like unreliable data environments. Notably, it approaches the upper-bound performance of centralized systems under ideally reliable data environments. The decentralized infrastructure enables secure and trustworthy scoring management, achieving approximately 56\% marginal cost savings through batched update operations. Our code and system are open-sourced at github.com/yining610/Reliable-dRAG.
1 INTRODUCTION
dRAG lets language models retrieve from owner-controlled sources while addressing the reliability differences among decentralized sources. It dynamically scores sources and records those scores on blockchain, and experiments report stronger performance than centralized RAG in unreliable settings.
1 INTRODUCTION
- dRAG retrieves documents directly from sources maintained and controlled by individual data owners, who choose what to share and which retrieval policies to implement.This design addresses centralized data-management costs and privacy concerns as the number of sources grows.Source: §1 · p. 1
- Its scoring mechanism evaluates sources from the quality of responses they help generate and prioritizes high-quality sources during retrieval.The stated aim is to improve retrieval accuracy and response quality in noisy data environments.Source: §1 · pp. 1–2
- Blockchain smart contracts manage reliability scores and logs, producing transparent, tamper-proof records that participants can verify without a central authority.Contract execution and verification are performed collectively by most nodes in the blockchain network.Source: §1 · pp. 1–2
- dRAG offers owner-controlled retrieval augmentation and continuously updated scores that can encourage owners to monitor and improve their sources.The system is described as applicable to settings where data cannot be centrally managed.Source: §1 · p. 2
- The experiments simulate unreliable sources by injecting incorrect answers into Wikipedia documents used to answer Natural Questions queries.Documents are polluted to varying reliability levels by replacing ground-truth tokens with incorrect answers.Source: §1 · p. 2
- dRAG outperforms centralized systems in unreliable data environments and approaches the performance of centralized systems with fully reliable sources.The introduction also reports that performance improves as more user queries are processed.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORK AND BACKGROUND
The related work describes the limits of centralized RAG and distinguishes dRAG from existing decentralized approaches. dRAG uses smart contracts as a lightweight trust layer for reliability scoring rather than coordinating knowledge propagation or training.
2.1 Centralized Retrieval Augmented Generation
- Centralized RAG faces scalability, data-management, privacy, and governance challenges as data volumes and domains increase.The section attributes these constraints to single-index, monolithic control and diverse-source management.Source: §2.1 · p. 2
- dRAG distributes retrieval across independent data providers to address centralized data-management limitations.The related-work section also presents distributed retrieval as improving scalability and data management.Source: §2.1 · pp. 2–3
2.2 Decentralized Retrieval Augmented Generation
- Unlike decentralized approaches involving federated training or parameter sharing, dRAG operates at inference time without training overhead and logs source contributions on blockchain.The section characterizes this as a lightweight approach to quality control.Source: §2.2 · p. 3
- dRAG uses smart contracts to dynamically evaluate and prioritize sources by reliability, avoiding complex validator committees or centralized content reindexing.The contracts provide a transparent, tamper-proof trust layer over decentralized retrieval.Source: §2.2 · p. 3
3 SYSTEM ARCHITECTURE
dRAG consists of independently maintained data sources, an LLM service, and a blockchain network that manages source reliability. Its workflow samples sources, retrieves and reranks documents, generates an answer, and updates scores using feedback.
3.1 System Components
- The architecture comprises decentralized data sources, an LLM service, and a decentralized blockchain network.Owners maintain sources, while the blockchain network hosts the smart contract scoreboard and the LLM service interacts with both sources and contract.Source: §3.1 · p. 3
3.2 System Workflow
- For each query, the LLM service samples sources using reliability evaluations, retrieves relevant documents, reranks them using reliability scores, and generates an answer.Sampling can be customized; dRAG describes a process based on each source’s usefulness to LLM performance.Source: §3.2 · p. 4
- User feedback updates reliability scores on the blockchain, with automatic exact-match evaluation also supported when ground-truth answers are available.The workflow assumes the LLM service can access sources through standard-compliant interfaces.Source: §3.2 · pp. 3–4
4 EVALUATING DATA SOURCES RELIABILITY
dRAG estimates sentence contributions to generated responses, aggregates them into source-level reliability updates, and uses usefulness for source sampling and reliability for reranking.
4.1 Sentence Importance Evaluation
- Without ground-truth answers, MC-Shapley estimates each sentence’s marginal contribution across sampled sentence subsets, using response cosine similarity as its utility function.This formulation is described as more generalizable but computationally more costly.Source: §4.1 · p. 4
- With ground-truth answers, conditional V-information measures how much conditioning on a sentence reduces predictive uncertainty; trained evaluators improve efficiency but reduce out-of-domain generalizability.Sentence scores are aggregated by arithmetic mean for Shapley scores and by maximum for entropy-based scores.Source: §4.1 · p. 4, §4.2 · p. 4
4.2 Data Source Reliability Update
- For grounded responses, source reliability increases or decreases by the document score according to correctness; ungrounded responses instead reduce usefulness while leaving reliability unchanged.These additive updates accumulate evidence across interactions.Source: §4.2 · p. 5
- dRAG samples sources by usefulness, reranks retrieved documents using normalized source reliability, and updates source scores from the retrieved documents’ sentence-importance estimates.The reranking score blends the original score with normalized reliability; α controls reliability’s influence.Source: §4 · p. 4, §4.2 · p. 5
5 ON-CHAIN RELIABILITY MANAGEMENT SYSTEM WITH SMART CONTRACT
dRAG manages source scores through blockchain smart contracts and signed feedback logs, aiming to make score updates transparent, tamper-resistant, and traceable to queries. The design also describes optional replay protection and asynchronous feedback when ground truth is delayed.
5.1 On-Chain Trustworthy Reliability Management with Smart Contract
- The blockchain acts as a public bulletin board for initialized source scores, while smart contracts handle score initialization, retrieval, updates, and auditable logs.The design seeks public auditing and protection against score tampering without a single managing party.Source: §5.1 · p. 5
- Each sampled source signs a query-bound digest of the query and sampled-source scores; the contract batch-verifies signatures and feedback-log information before updating scores.The feedback log includes the query, importance-evaluation summary, and reported scoring-state hash.Source: §5.1 · p. 6
- Replay protection is optional: pre-log each query and sampled-source list as unused, then mark the log used after the first feedback to invalidate later updates.This is proposed to prevent reusing signatures to maliciously update scores.Source: §5.1 · p. 6
5.2 Auditing Reliability Updates
- Score-update events record information linking updates to the original query, enabling anyone to audit the reliability-update history and reproduce queries for investigation.Event data is linked to the blockchain transaction and retrievable by gateway nodes.Source: §5.2 · p. 6
- For delayed ground truth, users can submit feedback with an undetermined importance report, then report ground truth on-chain later to update reliability scores.Sentence-level importance can be evaluated without ground truth in the meantime.Source: §5.2 · pp. 6–7
6 EXPERIMENTS AND RESULTS
Across simulated six-source pollution settings, dRAG outperforms centralized RAG on unreliable data and approaches centralized performance on reliable data under token-level pollution. Its results vary with source selection and pollution strategy, while batched blockchain updates reduce marginal costs.
6 EXPERIMENTS AND RESULTS
- Experiments use two Llama instruction models and six sources spanning 0%–100% pollution under token- and document-level constructions.The two environments use token-level noise replacement and disjoint document-level source partitions.Source: §6 · p. 7
6.1 Centralized versus Decentralized RAG System
- In unreliable settings, dRAG improves across queries as source A’s use rises from ~40% to >80%, while B–D fall from ~60% to <20%.Query bins contain 100 queries each; source A’s reliability score grows as it contributes to correct answers.Source: §6.1 · pp. 7–8, Figure 5 · p. 8
- dRAG consistently outperforms centralized RAG on unreliable data across configurations and reaches comparable performance to centralized systems on reliable data under token-level pollution.Table 1 reports average accuracy after 500 warmup queries for dRAG.Source: Table 1 · p. 8, §6.1 · p. 8
- Document-level pollution yields lower accuracy because sources cover only 1,066 of 3,197 documents, risking irrelevant retrieval when highly reliable sources are promoted.The authors leave handling incomplete and unreliable sources to future work.Source: §6.1 · p. 8
6.2 Sensitivity Analysis and Score Validation
- Increasing α strengthens reliability-driven filtering toward the upper bound, but α=1.0 overlooks semantic relevance and drops accuracy to approximately 34%.The sensitivity test uses MC-Shapley with LLAMA-3.2-3B-INSTRUCT on token-level polluted data.Source: Figure 6 · p. 9, §6.2 · p. 9
- Increasing N generally raises accuracy toward the upper bound, but N=6 drops performance when usefulness-based sampling is disabled.N controls how many data sources are sampled based on usefulness scores.Source: Figure 6 · p. 9, §6.2 · p. 9
- MC-Shapley and RORA sentence-importance score entropy decreases as token pollution rises, reflecting increasingly uniform low scores across sentences.The authors interpret the entropy pattern as sensitivity to data-quality degradation.Source: Figure 6 · p. 9, §6.2 · p. 10
6.3 System Cost Evaluation
- Batching cuts marginal gas cost by 55.8%, from 2.6 cents per update at n=1 to 1.1 cents at n=20.Actual gas consumption also depends on query length and other factors.Source: Table 3 · p. 10, §6.3 · p. 10
7 CONCLUSION AND FUTURE WORK
dRAG dynamically updates source-reliability scores and secures them on blockchain; controlled experiments report better performance than centralized systems in unreliable environments and comparable performance to fully reliable environments. The conclusion identifies document-level pollution and unknown score-convergence dynamics as remaining challenges.
7 CONCLUSION AND FUTURE WORK
- Across models, pollution strategies, and scoring methods, dRAG consistently outperforms centralized systems in unreliable environments and achieves performance comparable to fully reliable environments.Source: §7 · p. 10
- dRAG dynamically computes and updates each source’s reliability score and secures scores on blockchain to support quality control and trust among owners.Source: §7 · p. 10
- Under document-level pollution, performance falls considerably short of the centralized fully reliable upper bound because sources cover only part of the query space.The authors suggest query-conditioned gating to choose reliability-based reranking or preserve relevance ordering.Source: §7 · p. 10
- How many queries are needed for reliability scores to reach the performance upper bound, and which factors shape convergence, remain open questions.The authors identify query diversity, source count, and source quality as factors for future study.Source: §7 · p. 10