PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems

Ramon Gonzalez, Antonio Diaz

arXiv:2609.07075v1cs.AI

TL;DR

Answer-only evaluation can miss contradictory retrieved evidence, and similarity scores do not establish factual agreement. HCF audits corpus, retrieval, and answer consistency with source-linked conflict diagnostics and answer explanations; across controlled evaluations, it exposed conflicting context even when answers matched ground truth. Its evidence is bounded by small constructed collections, one retained run per condition, and non-independent evaluation of ACS.

  • Problem

    Answer correctness and retrieval similarity alone do not reveal whether retrieved evidence is factually consistent or contains contradictions.

  • Method

    HCF provides a post-hoc, model-agnostic audit of corpus, retrieval, and answer consistency, using atomic facts, provenance, and answer-level evidence comparisons.

  • Results

    Mean ACS was 0.9844, 0.8780, 0.9320, and 0.9244 for Corpora A–D, respectively; Corpus B had the lowest mean ACS despite the highest mean SSµ of 0.8843.

  • Takeaways & Limitations

    HCF makes supporting and contradictory evidence inspectable and attributable, including cases where the answer matches ground truth but the retrieved context contains a conflicting claim.

  • Takeaways & Limitations

    The controlled collections are small and constructed, with one retained run per question and condition, and ACS lacks independent validation beyond the single evaluator.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its reference while the context that produced it contains a direct contradiction, leaving the contested evidence invisible to answer-only review and retrieval relevance scores. This paper presents the Hierarchical Consistency Framework (HCF), a post-hoc, model-agnostic audit of three distinct levels of a RAG process: the knowledge corpus, the final retrieved context, and the generated answer. HCF represents corpus conflicts as source-linked atomic facts, thereby identifying the documents responsible, and returns each Answer Consistency Score (ACS) with an explanation of supporting and contradictory contextual statements. We evaluate HCF on several controlled corpora spanning five domains and 100 query-corpus instances. A human evaluator compares every generated response with its supplied ground-truth response. The results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity has the lowest mean ACS, while a structurally degraded corpus performs worse at corpus level but better at answer level. Most importantly, HCF identifies contradictory retrieved evidence in several cases where the answer still matches the ground truth. HCF does not certify factual truth; it makes the evidence supporting and challenging an answer inspectable and attributable.

I. INTRODUCTION

HCF addresses the difficulty of diagnosing unreliable RAG outputs by auditing corpus, retrieval, and answer consistency separately. It provides post-hoc, model-agnostic diagnostics that localize failures and make evidence conflicts inspectable.

  • RAG supplements a parametric generator with dynamically retrieved evidence from an external corpus.
  • Unreliable RAG outputs may originate from degraded corpora, contradictory retrieval, or failure to ground answers in context.
  • Dense retrieval scores measure query–passage relevance, not factual agreement across sources, so similarity alone cannot diagnose conflicting evidence.
  • HCF provides post-hoc, model-agnostic diagnostics across Corpus Consistency, Retrieval Consistency, and Answer Consistency.Its retrieval metrics and Answer Consistency Score also provide explanations that help localize failures.
  • The framework extends corpus-aware and explainable RAG evaluation by using decomposable diagnostics for document quality, cross-document conflict, retrieval dispersion, and answer grounding.
  • HCF combines complementary corpus dimensions rather than treating corpus consistency as a single undifferentiated quality property.

1) Intra-Document Metrics:

The corpus-level audit measures structural document quality and cross-document factual agreement. Its inter-document component extracts source-linked atomic facts, indexes them by entity and relation, and records conflict provenance.

  • Intra-Document Metrics:: Intra-document quality evaluates clarity, internal consistency, completeness, information density, and coherence, but does not estimate factual truth.
  • Inter-Document Metrics:: Inter-document consistency detects factual divergences across document boundaries using atomic fact extraction rather than isolated quality or similarity assessments.
  • Inter-Document Metrics:: Each atomic fact triple records a source document, named entity, relation or attribute, and asserted value as one independently verifiable claim.For example, a document stating that Spain’s capital is Madrid yields a corresponding source-linked triple.
  • Inter-Document Metrics:: The corpus fact index groups document–value pairs by normalized entity–relation keys to compare claims across documents.Normalization lowercases and removes diacritics before values are treated as identical.
  • Inter-Document Metrics:: The minimal normalization strategy may miss conflicts caused by surface variation such as “Spain” versus “Kingdom of Spain.”Richer embedding-based entity clustering is left for future work.
  • Inter-Document Metrics:: Conflict provenance identifies every document contributing to a detected cross-document conflict, supporting the framework’s explainability layer.

3) Unified Corpus Consistency Metric:

Unified Corpus Consistency combines intra-document structural quality with inter-document factual agreement in a bounded, decomposable score. The aggregation is configurable, and the experiments weight both dimensions equally.

  • 3) Unified Corpus Consistency Metric:: Corpus Consistency combines the two corpus dimensions into a single corpus-level diagnostic.
  • 3) Unified Corpus Consistency Metric:: CC ranges from 0 to 1, with CC = 1 denoting a fully consistent corpus and CC = 0 a completely degraded one.
  • 3) Unified Corpus Consistency Metric:: The linear aggregation keeps each quality dimension independently interpretable and auditable, allowing low CC to be attributed to structural quality, factual conflict, or both.
  • 3) Unified Corpus Consistency Metric:: The experiments set wc = wn = 0.5, treating intra-document and inter-document quality as equally important without domain-specific prior knowledge.
  • 3) Unified Corpus Consistency Metric:: The IeDCD computation extracts atomic facts, builds a corpus fact index, detects conflicting assignments, and returns a consistency score with provenance sets.

B. Level 2: Retrieval Consistency

Retrieval Consistency characterizes the relevance and score dispersion of evidence selected for a query using three indicators derived from query–chunk similarity scores. These indicators must be interpreted jointly with corpus consistency and retrieved text because score dispersion alone does not establish consistency.

  • Retrieval consistency indicators: Retrieval Consistency uses three indicators derived from query–chunk similarity scores.The indicators are top retrieval similarity, mean retrieval similarity, and retrieval similarity variance.
  • Retrieved evidence set: R(q) denotes the retrieved chunks for query q, with k equal to the number of retrieved chunks.The retriever assigns each chunk a normalized similarity score in [0, 1].
  • Retrieval consistency indicators: Top Retrieval Similarity, SS1(q), measures the similarity of the highest-ranked retrieved chunk.It captures the strength of the single most relevant retrieved evidence source.
  • Retrieval consistency indicators: Mean Retrieval Similarity, SSµ(q), measures average semantic agreement across the retrieved evidence set.Higher values indicate stronger overall retrieval coherence.
  • Retrieval consistency indicators: Retrieval Similarity Variance, SSσ(q), measures dispersion of similarity scores rather than consistency by itself.High values indicate heterogeneous relevance, while unusually low values can also signal contradictory but topically similar chunks compressed into one embedding neighborhood.

C. Level 3: Answer Consistency

Answer Consistency evaluates whether a generated answer remains grounded in retrieved evidence and aligned with the original query. Its evaluator considers query alignment, evidence support, and absence of contradiction, but ACS measures grounded agreement within the pipeline rather than factual truth.

  • Answer Consistency: Answer Consistency measures agreement among the generated answer, retrieved evidence, and user query.It evaluates the synthesis stage of the RAG process.
  • Answer Consistency: J evaluates query alignment, evidence support, and absence of contradiction for query q, answer a, and retrieved evidence R(q).The evaluator returns a value in [0, 1].
  • Evaluator: The evaluator J may be rule-based, learned, or hybrid.Its experimental instantiation is specified separately.
  • Scope: ACS is not a probabilistic confidence estimate or a measure of factual truthfulness.A highly consistent answer may still be factually incorrect when the retrieved evidence is incorrect.

A. Experimental Platform

The experiments use Mentomy as a platform while treating HCF itself as platform-agnostic, with separate LLM task instructions for assessment, extraction, generation, and evaluation. Four controlled corpora span five domains and vary conflict and structural degradation conditions.

  • Experimental platform: Mentomy provided ingestion, retrieval, access control, answer generation, and evaluation workflows, while HCF definitions and operators remained platform-agnostic.Mentomy was the experimental platform only.
  • Operator instantiation: Generic operators Q, F, and J were implemented through task-specific instruction sets supplied to Mistral Small 4.LLM instantiation was an experimental choice rather than an HCF requirement.
  • Operator roles: Q assessed document clarity, internal consistency, completeness, information density, and coherence without external fact-checking; F extracted source-linked atomic facts; J assessed answer grounding and contradiction.The operators were applied with extraction caps to bound latency.
  • Implementation: All three operators and answer generation used Mistral Small 4 at temperature 0.1, with separate instructions isolating their roles.The model was released on 16 March 2026 and has 119 billion total parameters.
  • Retrieval implementation: Retrieval used Pinecone dense-vector search, multilingual-e5-large embeddings, reranking, and retention of 10 passages after 15 candidates were selected.The index dimension was 1,024, and the embedding model accepted at most 507 input tokens per sequence.
  • Controlled corpora: Four controlled corpora were constructed across physics, medicine, history, geography, and biology.The corpus conditions included a clean baseline, conflict-injected documents, structural degradation, and combined degradation.
  • Controlled corpora: Corpus D combined clean, contradictory, structurally degraded, and fully corrupted documents to represent unmanaged corpus growth.It contained 20 documents across all degradation conditions.

D. Rationale for a Controlled Corpus

The study uses controlled corpora to obtain exhaustive factual ground truth and isolate degradation modes, enabling paired comparisons across identical queries. Human reference matching is evaluated separately from ACS, while corpus-level diagnostics distinguish integrity and cross-document divergence.

  • Why controlled corpora: Controlled corpora provide ground truth for conflicting entity–relation pairs, asserting documents, canonical values, and source-linked provenance.Complete factual ground truth is generally unavailable for organic corpora.
  • Experimental isolation: The four corpora form a 2 × 2 factorial design over low and high inter-document conflict and structural degradation.Corpus D additionally varies corpus size, while the controlled design eliminates confounding between the two consistency dimensions.
  • Baseline construction: Baseline documents were generated and reviewed by multiple models for factual errors, internal inconsistencies, unclear passages, and deviations from canonical sources.Corrections flagged by either reviewer were applied before corpus admission.
  • Evaluation design: The study evaluates 100 query–corpus instances by issuing the same 25 queries to each of four corpus conditions.Fixed queries enable within-query paired comparison across degradation conditions.
  • Query design: Queries include probes for manipulated entity–relation values, controls, and multi-part synthesis, while retrieved context remains determined by retrieval, reranking, and authorization.The query labels describe intended probes rather than guaranteed retrieved content.
  • Reference assessment: One human evaluator compared every generated response with its supplied ground-truth response, separately from ACS.Exact and semantically equivalent answers were accepted, while missing or materially different content was retained in the result register.
  • Corpus diagnostics: Corpus-level diagnostics distinguish degradation conditions through ingestion measurements and calculated Corpus Consistency.A and B had similar integrity, but only B had flagged divergences; C had low integrity without detected cross-document divergences, and D had the largest flagged-key count.

B. Evaluation of Atomic Fact Extraction

The evaluation measures extraction coverage, semantic correspondence, and query-level answer and retrieval diagnostics across controlled corpora. Results show uneven extraction coverage, strong but imperfect correspondence, and divergences between semantic relevance and answer consistency.

  • Atomic fact extraction: 34 of 40 selected reference facts were recovered (85.0%), with complete recovery in geography and biology but four of six omissions in history.The assessment counted semantically equivalent expressions as recovered.
  • ER-pair correspondence: 24 of 27 selected clean-side entity–relation records matched corresponding records in modified documents, yielding an 88.9% correspondence rate.History and geography achieved complete correspondence, while physics, medicine, and biology each had one unmatched record.
  • Answer consistency: Median ACS was 1.00 in every corpus, but distributions differed by question type and corpus, with Corpus B showing the largest targeted-question reduction.Corpus C had lower values for several synthesis questions, while Corpus D showed reductions in targeted and synthesis questions.
  • Query-level diagnostics: Corpus B had the highest mean SSµ (0.8843) but the lowest mean ACS (0.8780), showing that semantic relevance and evidence consistency can diverge.Retrieved chunks can remain strongly related to a question while disagreeing about the requested fact.
  • Query-level diagnostics: Mean ACS was 0.9844, 0.8780, 0.9320, and 0.9244 for Corpora A–D, respectively, while SS1 means remained high at 0.8928–0.9605.A strong top hit therefore does not establish that the remaining retrieved context is compatible.
  • Answer consistency: In four Corpus B cases, answers matched the ground truth while ACS identified conflicting values or dates in the final retrieved context.The accompanying explanations directed readers to the competing claims, making the reduced scores actionable.

VI. DISCUSSION

The discussion separates answer correctness from evidential consistency: HCF exposes contradictory retrieved claims even when answers match references, while staged diagnostics avoid assuming that upstream scores determine answer consistency.

  • Evidential setting: HCF’s central contribution is exposing evidence inconsistency that reference-only correctness assessment would miss.An answer-only check accepts the response, whereas ACS identifies an incompatible retrieved statement and directs reviewers to the competing source.
  • Limits of similarity: 0.8843 mean SSµ paired with 0.8780 mean ACS in Corpus B shows that high semantic relevance can coexist with conflicting evidence.Retrieved chunks may remain strongly related to the question while disagreeing about the requested fact.
  • Staged inspection: Corpus diagnostics distinguish structural integrity and indexed divergence before attributing failed comparisons to the answer generator.The staged organization also makes omissions and representation differences visible.
  • Limits of similarity: Corpus composition, ranking, and authorization jointly determine which evidence reaches the answer stage, so CC, retrieval variance, and ACS have no assumed monotonic relationship.A low variance can also arise trivially when only one chunk is retained.
  • Operational interpretation: Flagged cases should prompt source inspection rather than automatic declarations that answers are false.Threshold-based actions such as abstention or source repair require separately validated decision rules.

VII. CONCLUSIONS

HCF provides linked diagnostics for corpus, retrieval, and answer consistency, illustrating that correct answers can coexist with contradictory evidence. Its controlled results support decomposition and localization, not generalization or validated correctness probabilities.

  • Conclusions: HCF links corpus integrity, retrieval statistics, and answer–context consistency to localize where evidence limitations arise.Its model-independent formulation provides a common structure for examining the RAG process.
  • Conclusions: Correct answers can coexist with contradictory retrieved evidence, while similarity dispersion and corpus-level consistency do not determine answer consistency in the four tested conditions.HCF is therefore complementary to factual verification rather than a replacement for it.
  • Scope: 40 document placements across four configurations represent 20 document variants in five domains, but the small constructed collections cannot establish generalization to organic enterprise knowledge bases.The conditions differ in size and redundancy, with one retained run per question and condition.
  • Validation limits: A single human evaluator supports benchmark-match assessments but does not independently validate ACS.Independent adjudication of answer–context pairs is needed to assess ACS sensitivity, specificity, stability, and calibration.
  • Algorithmic limits: Equivalent multilingual expressions, multivalued relations, and omitted temporal qualifiers can distort candidate-divergence counts.The semantic correspondence review is broader than the minimal runtime comparator and is not proof of automated semantic matching.
  • Future research: Future validation should vary prompts, score provenance, chunking, and authorization independently, repeat runs, and compare HCF with reference-free metrics.Larger corpora along controlled consistency gradients are needed to test whether upstream degradation propagates downstream.
  • Reproducibility: The released dataset contains Corpora A–D, 25 evaluation questions, and supplied ground-truth responses, but excludes Mentomy’s proprietary platform.Main-text extracted records and answer excerpts are translated or summarized for readability.

APPENDIX A DATASET DOCUMENTATION

Appendix A documents the public evaluation dataset: Corpora A–D, 25 questions, and their ground-truth responses, while excluding the proprietary Mentomy platform implementation.

  • Dataset contents: The public dataset contains the documents forming Corpora A–D, 25 evaluation questions, and their supplied ground-truth responses.The questions comprise 10 manipulated-relation targets, 10 controls, and 5 synthesis questions.
  • Dataset contents: Mentomy’s proprietary platform implementation is not included in the public dataset.The release therefore provides evaluation materials without the platform itself.
  • Reported results: The query-level results are organized in separate tables for Corpora A, B, C, and D.These tables report the corpus-specific question results.
Loading 2609.07075v1…