Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems

Balkrishna Giri; Md Toufique Hasan; Jussi Rasku; Muhammad Waseem; Pekka Abrahamsson

Source-linked AI summary

Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems

Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson

arXiv:2608.21095v1Aug 21, 2026cs.SEcs.AIcs.CLcs.CRcs.IR

TL;DR

RAG can treat relevant retrieved text as trustworthy even when adversaries have poisoned it. This paper introduces an Evaluation Agent that combines NLI verification, multi-signal poison detection, and a dampened Trust Index; it detects overt attacks well but remains limited by in-place edits, generator dependence, and domain shift.

Problem
RAG evaluation often measures faithfulness to retrieved context without verifying whether the context itself is trustworthy, leaving knowledge poisoning insufficiently addressed.
Method
The Evaluation Agent uses defensive middleware combining NLI factual verification, a five-signal poison detector, relevance-weighted aggregation, and a Trust Index with a high-contamination dampener.
Results
Across tested settings, overt poisoning was detected reliably, including 100% recall for instruction injection and 91% mixed accuracy with 100% precision, while in-place edits remained near-undetectable.
Takeaways & Limitations
The agent adds an online context-integrity check to RAG and can block overt unsafe-guidance injection in a secure-coding setting, while calibration remains important across models and datasets.
Takeaways & Limitations
The study measures poisoned-context detection before generation, not whether the LLM adopts injected misinformation, and stronger attacks require future end-to-end evaluation.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) outputs in external knowledge, but RAG systems usually trust whatever they retrieve, creating a Security-Reliability Gap: high semantic relevance does not guarantee factual truth. Adversaries exploit this through knowledge poisoning, inserting malicious documents to cause targeted misinformation. We propose an Evaluation Agent, middleware that combines Natural Language Inference (NLI) factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index T = 0.4 F + 0.35 C + 0.25 (1 - P ) with a non-linear dampener for high-contamination contexts. On TruthfulQA with Llama 3.3 70B, the agent reaches 91% accuracy and 100% precision, with 100% recall on instruction injection, while in-place edits, such as entity swaps, remain hard to detect. Across three LLMs the Trust Index stays discriminative, with a Receiver Operating Characteristic Area Under the Curve (ROC-AUC) of 0.73 to 0.81; generation style matters more than model size, and per-LLM threshold calibration restores baseline competitive accuracy, whereas a weaker FEVER result shows that cross-dataset generalization requires domain-specific calibration. In a software-engineering use case, a secure-coding assistant over guidance from the Open Worldwide Application Security Project (OWASP) Top 10 and the Common Weakness Enumeration (CWE), the agent reliably blocks instruction injection of unsafe advice (F1 92%), while contradiction and subtle semantic weakening remain hard. Throughout, the agent measures detection of poisoned context before generation, not whether the LLM adopts the injected misinformation. We release the proposed approach, attack generator, and experimental artifacts at the link: https://github.com/GPT-Laboratory/TrustworthyRAG.

I. INTRODUCTION

RAG improves grounding but can inherit poisoned retrieved documents, creating a gap between semantic relevance and factual trust. The Evaluation Agent addresses this gap with online context screening, NLI verification, poison detection, and an interpretable trust layer evaluated across benchmarks, models, attacks, and secure coding.

Motivation: ≈0.0002% poisoned passages can drive attack success to roughly 90% in a corpus of millions.This exposes why relevance-based RAG evaluation can miss compromised outputs.
Approach: The Evaluation Agent screens retrieved context before generation and fuses NLI factual verification, five-signal poison detection, and cross-document consistency.It is defensive middleware designed to add an online trust layer inside the inference loop.
Evaluation scope: The study evaluates TruthfulQA and FEVER across three LLMs and four attack strategies, then tests a secure-coding assistant using OWASP Top 10 and CWE guidance.The research questions target detection, NLI-based trust, and resilience across generators, domains, and attacks.
Findings: 100% recall on instruction injection and 91% accuracy with 100% precision on TruthfulQA mixed attacks demonstrate strong detection of overt poisoning.In-place edits, including entity swaps and subtle weakening, remain near-undetectable.
Implications: The approach adds an online context-integrity check to RAG, while identifying surface-signal and cross-dataset generalization boundaries.The paper reports that generation style and dataset shift affect trust scoring and that stronger world-knowledge verification is needed for some attacks.

II. BACKGROUND AND RELATED WORK

Prior RAG evaluation emphasizes retrieval relevance and answer faithfulness, while attacks can poison retrieved knowledge or inject unsafe instructions. The paper positions its Evaluation Agent as a joint context-and-answer trust layer, including for secure-coding RAG.

Evaluation gap: Retrieval metrics such as top-k accuracy and MRR measure relevance, but a highly relevant poisoned document can still be unsafe.TruthfulQA and NLI-based benchmarks broaden evaluation toward factual consistency, but do not by themselves establish corpus integrity.
Threats: Indirect prompt injection and inference-corpus poisoning let adversaries place malicious content into retrieved context.Optimized adversarial passages can be designed to enter retrieved sets for many queries.
Positioning: Existing defenses filter suspicious passages or aggregate isolated generations, often requiring retriever-side assumptions or multiple generations.The proposed agent instead scores the retrieved set and generated answer jointly without modifying the retriever or generator.
Software-engineering setting: Secure-coding RAG makes its guidance corpus a poisoning target because developers may over-trust assistants that generate insecure code.The use case draws on OWASP Top 10 and CWE guidance.
Threat model: The threat model assumes a black-box adversary can insert or modify retrieval documents but cannot access model weights, user prompts, or the Evaluation Agent.The attacker seeks misinformation or unsafe recommendations in third-party, web-sourced, or community-contributed content.
Proposed system: The middleware orchestrates NLI verification, poison detection, and Trust Index calculation to emit a numeric score, trust level, and warnings.Poison and consistency checks run before generation, while NLI compares retrieved documents with the generated answer afterward.

A. Factual Verification via NLI

The NLI module estimates factuality by comparing each retrieved document with the generated answer, while the poison detector uses complementary signals to identify adversarial context. This separation reserves strong negative evidence for poison detection rather than raw contradiction scores.

NLI factual verification: The NLI verifier treats each retrieved document as a premise and the generated answer as a hypothesis, producing entailment, neutral, and contradiction probabilities.Factuality aggregates entailment over documents with meaningful entailment; otherwise it returns a 0.5 inconclusive baseline.
NLI limitation: ≈0.99 contradiction scores can occur for both genuine contradictions and unrelated text, making raw contradiction an unreliable negative signal.The design therefore reserves strong negative signals for the dedicated poison pathway.
Multi-signal detection: The poison detector combines linguistic, structural, intra-document, cross-document, and semantic-outlier signals.These target override phrases, formatting artifacts, internal contradictions, inconsistent insertions, and embedding deviations.
Aggregation: Relevance weighting prevents low-relevance suspicious neighbors from dominating the batch poison verdict.The weighting uses each document’s retrieval similarity when available.
Boundary: In-place value substitutions that preserve surface form fall outside the detector’s signal set by construction.The paper identifies this as a quantified limitation of surface-based detection.

C. The Trust Index

The Trust Index combines factuality, consistency, and inverse poison probability, then dampens scores under high contamination. Its weighting reflects differing signal reliability, but factuality remains dependent on the generating LLM’s style.

Trust Index: T = 0.4 F + 0.35 C + 0.25 (1 − P) combines factuality, consistency, and inverse poison probability into a trust score.The default weights sum to one and place the greatest weight on NLI factuality.
Weighting rationale: The weights prioritize NLI entailment, then cross-document agreement, while assigning the lowest weight to heuristic poison signals.This design limits false-positive influence, and the operating point is stable under moderate weight perturbations.
Non-linear dampening: Ppoison > 0.70 activates a multiplicative dampener because a linear score can remain above τ=0.5 under high contamination.The dampener produces Tfinal = T · d(P).
Worked example: For T=0.59 and P=0.9, d(0.9)=0.733 yields Tfinal=0.43 < τ, flagging the contaminated response.The multiplier is continuous, bounded at d(1.0)=0.6, and inactive for clean contexts.
LLM dependency: Generation style affects factuality scores because hedged prefixes lower entailment even for correct answers.Consequently, the threshold and weights function as effectively LLM-specific hyperparameters.

IV. EXPERIMENTAL DESIGN

The experiments use paired clean and poisoned benchmark queries, multiple attack strategies and model configurations, plus a secure-coding corpus curated from OWASP and CWE guidance.

Datasets: The study evaluates TruthfulQA and FEVER, plus a 40-rule secure-coding knowledge base curated from OWASP Top 10 and CWE guidance.The secure-coding rules paraphrase and cross-check normative source guidance and cover major OWASP categories.
Protocol: Each run contains 50 clean queries and 50 queries against a corpus with 30% poisoned documents.The poisoned condition uses the same queries as the clean condition.
Attack strategies: The protocol tests contradiction, instruction injection, entity swap, and subtle manipulation attacks, including a mixed strategy setting.Entity swaps replace entities or numbers in place, while other strategies append or alter textual content.
Models and metrics: Generation uses Llama 3.3 70B or Qwen 3.5 35B, with 384-dimensional or 1024-dimensional document embeddings and top-K FAISS retrieval.Reported metrics include accuracy, precision, recall, F1, and trust-score separation Δ.
Ground truth: Ground truth labels samples by whether the aligned source document was modified, even when that poisoned document is absent from the retrieved context.This document-level labeling is stricter than a retrieval-grounded label.

V. RESULTS

This section evaluates detection performance, component contributions, configuration robustness, and a secure-coding RAG application.

The results section evaluates detection, component contributions, robustness across configurations, and secure-coding RAG performance.

A. Detection Performance and Per-Strategy Hierarchy

Detection is strongest for overt instruction injection and weakest for in-place or subtle edits, while component ablations expose a precision–recall trade-off in the Trust Index.

Overall detection: 91% accuracy and 100% precision are achieved on the primary TruthfulQA mixed run, with 40% recall and F1 57.1%.The run uses Llama 3.3 70B, MiniLM, and K=5; trust-score separation is Δ=0.225.
Per-strategy hierarchy: 100% recall is achieved for instruction injection, compared with 53.3% for contradiction, 20% for subtle manipulation, and 0% for entity swaps.The tested setting contains only 15 poisoned samples per run, so per-strategy confidence intervals are wide.
Component ablation: NLI factual verification alone yields 0% recall because it scores entailment against the generated answer rather than adversarial intent.The ablation recomputes verdicts by re-weighting stored factuality, consistency, and poison scores at τ=0.5.
Component ablation: Fusing factuality, consistency, and poison signals reaches 100% precision, while the poison detector alone reaches 53.3% recall and F1 64.0% at 80% precision.The fused system trades recall for zero false positives, with factuality and consistency supplying precision and the poison pathway supplying recall.
Weight sensitivity: Perturbing each Trust Index weight by up to ±0.10 keeps accuracy at 90.0–91.0%, F1 at 54.5–57.1%, and precision at 85.7–100%.Across 125 renormalized settings, no perturbation outperforms the default weights.

C. LLM and Embedding Sensitivity

LLM choice strongly affects the Trust Index, while embedding dimensionality has little effect for concise-output models. Threshold-independent discrimination remains above chance across all tested LLMs, and per-LLM calibration improves comparability.

LLM and embedding effects: 91% accuracy and 100% precision held across both Llama configurations regardless of embedding model, while Qwen reached 71% with 25 to 28% precision.Qwen’s mean clean trust score was ≈0.64 versus ≈0.83 for Llama, producing 21 to 23 false positives among 85 clean samples.
LLM and embedding effects: Embedding dimensionality was near-irrelevant for a concise-output LLM, whereas LLM choice dominated detection performance.The comparison motivates treating generation style as the primary sensitivity rather than embedding configuration.
Run-to-run stability: 90.6±0.5% accuracy and 56.1±1.3% F1 were observed over five repeats, while instruction detection remained invariant at 99.0±0.0%.The low variance indicates that verdicts were driven by retrieved evidence rather than one generation’s surface wording.
Threshold-independent performance: ROC-AUC reached 0.81 for Llama, 0.79 for Mistral 7B, and 0.73 for Qwen, showing discrimination above chance across all three LLMs.The fixed τ=0.5 operating point understates the Trust Index for Qwen.
Per-LLM calibration: Qwen accuracy rose from 65.5% to 74.5% after per-LLM threshold calibration by reducing false positives.Optimal thresholds differed across Llama, Mistral, and Qwen, confirming that τ is LLM-specific.

E. Retrieval Depth and Overhead

Increasing retrieval depth from K=3 to K=5 leaves the mixed-strategy outcome unchanged, while K=3 achieves the same detection result at roughly half the NLI cost. The evaluation adds substantial per-sample latency that favors batch or asynchronous deployment.

Retrieval depth: 91% accuracy, 100% precision, 40% recall, and F1 57.1% remained unchanged when retrieval depth increased from K=3 to K=5.The reported change in ∆, 0.240→0.225, was within noise.
Retrieval depth: K=3 achieves the same mixed-strategy result at roughly half the NLI cost of K=5.Entity-swap and subtle attacks leave no textual signal at either retrieval depth, setting the recall ceiling by attack difficulty.
Overhead: ≈14.7 s per sample, or ≈17× baseline RAG overhead, is added by evaluation at K=5.Up to 20 CPU NLI passes dominate the cost, which grows linearly with K and is independent of corpus size.
Overhead: The latency profile suits batch settings, with asynchronous or batch-mode gating proposed for latency-sensitive deployments.GPU inference is projected to reduce evaluation below 2 s.

F. Cross-Dataset Generalization

The agent transfers to a secure-coding setting for overt unsafe-guidance injection, but cross-dataset performance and subtle semantic attacks expose important generalization limits.

Cross-dataset generalization: FEVER reaches 73% accuracy versus an 85% baseline, with separation collapsing to ∆=0.035 and requiring dataset-specific calibration.The result bounds external validity of the TruthfulQA findings despite adequate NLI context.
Secure-coding generalization: 92.3% F1 shows the agent detects injected malicious directives in the secure-coding assistant and blocks unsafe advice before it reaches developers.The use case retrieves from 40 secure-coding rules drawn from OWASP Top 10 and CWE.
Secure-coding generalization: 58% recall for entity swaps in secure coding exceeds 0% on open-domain TruthfulQA because structured inconsistencies can activate NLI signals.Contradiction and subtle manipulation still achieve 0% recall in the secure-coding setting.
Attack visibility: The agent reliably detects overt poisoning, but in-place edits and subtle weakening remain near-undetectable because they lack visible attack artifacts.The reported limitation is architectural rather than a tuning failure; external world knowledge is suggested for these attacks.
Operational trade-offs: Relevance-weighted aggregation mitigates but does not eliminate false-positive spillover when clean queries retrieve poisoned neighbors.Such elevated poison probability is security-correct for a contaminated environment but costly for precision.
Operational trade-offs: The agent can operate as an asynchronous workflow gate at approximately 17 seconds per query, with human review of flagged retrievals.The proposed placement is code review or continuous integration rather than the edit loop.
Validity boundaries: Reported recall is a conservative lower bound because document-level ground truth counts poisoning as missed when the document never enters the top-K context.Experiments also use controlled sample sizes and one curated secure-coding domain.

VII. CONCLUSION AND FUTURE WORK

The paper concludes that an Evaluation Agent can add an online trust layer to RAG, while identifying calibration, attack coverage, and end-to-end adoption measurement as necessary next steps.

Conclusion: The Evaluation Agent combines NLI factual verification, multi-signal poison detection, and a dampened Trust Index as a RAG trust layer.The conclusion reports clear poisoning detection and blocking of injected unsafe advice in the secure-coding use case.
Boundaries and future work: In-place edits are harder to detect, NLI trust scores depend on the generating LLM, and FEVER shows dataset-specific calibration is important for generalization.Future work includes stronger poisoning attacks, cross-LLM calibration, and world-knowledge sources for entity swaps.
Boundaries and future work: Future evaluation should measure whether poisoned guidance changes generated artifacts or developer decisions, rather than only detecting poisoned context before generation.The conclusion distinguishes detection from misinformation adoption or attack success rate.