CiteGuard-RAG: A Validation-Centered AI System for Evidence-Grounded Question Answering

1.1 Background and Motivation

• Legal question answering is vulnerable to hallucination, weak source traceability, and unreliable refusal, while legal corpora add structural and language complexity.The introduction argues that generated claims need to be checked against cited evidence rather than relying on retrieval alone.Source: §1.1 · p. 2

1.2.1 Retrieval Strategies

• Hybrid retrieval combines lexical exact-term matching with dense semantic similarity, but how retrieval strategies interact with grounding in legal RAG remains insufficiently understood.Legal rerankers and retrieval partitioning have shown gains in related settings.Source: §1.2.1 · p. 2

1.2.2 Grounding and Hallucination

• Retrieval alone does not ensure grounded generation: models can hallucinate from marginal evidence, underuse long contexts, or fail to integrate evidence.Prior constrained-generation and entailment-validation approaches address parts of this problem, but do not jointly impose grounding during generation and verify it at sentence level.Source: §1.2.2 · pp. 2–3

1.2.3 Evaluation

• End-to-end metrics can conflate retrieval errors with generation failures, and systematic evaluation of grounding reliability across system components remains lacking.Prior work also identifies robustness and rejection challenges and calls for fine-grained faithfulness and citation metrics.Source: §1.2.3 · p. 3

1.2.4 Research Gaps

• Four gaps remain: uncertain grounding under domain shift, undercharacterized retrieval effects, limited runtime sentence-level validation, and separate evaluation of interacting components.These gaps concern whether candidates are accepted, regenerated, or refused before delivery.Source: §1.2.4 · p. 3

1.3 Proposed Approach

• CiteGuard-RAG combines page-aware parsing, paragraph-aware chunking, hybrid retrieval, citation-constrained generation, sentence-level validation, and one-pass regeneration.Grounding is treated as runtime control: validation determines whether a candidate is accepted, regenerated, or replaced with a standardized refusal.Source: §1.3 · p. 3, Figure 1 · p. 4

1.4 Contributions

• The study connects retrieval, generation, validation, abstention, and regeneration, evaluates 400 questions across three settings, and analyzes reliability bottlenecks.The evaluation compares controlled housing-law performance with external domain- and task-shift behavior.Source: §1.4 · p. 4

2.1 System Overview

• Each substantive claim is checked against retrieved evidence and its citation before the output is delivered as a validated answer or standardized refusal.The validation result can accept the candidate, trigger one-pass regeneration, or replace it with a refusal.Source: §2.1 · p. 5
• The workflow retrieves hybrid-ranked evidence candidates and returns a standardized refusal when the evidence set is insufficient.The algorithm describes retrieval after constructing paragraph-aware evidence chunks.Source: §2.1 · p. 5
• A citation-constrained candidate is returned when validation passes; otherwise, validation feedback guides one regenerated candidate, which is returned only if it passes.If the regenerated candidate fails validation, the workflow returns a standardized refusal.Source: §2.1 · p. 5

2.2 Document Parsing and Chunk Construction

• Each extracted page stores normalized text, an internal PDF page index, and an optional human-visible printed page number.When printed numbering cannot be reliably detected, the system uses the canonical PDF page index to preserve traceability.Source: §2.2 · pp. 5–6
• Chunking prefers preserved paragraph boundaries and falls back to punctuation-based sentence boundaries when extraction has lost them.Adjacent units are aggregated under a maximum token budget with overlap to retain local context across chunk boundaries.Source: §2.2 · p. 6
• Chunk expansion is limited by page span, and navigation text, tables of contents, and very short segments are removed to reduce diffuse evidence.Final chunks retain text, a chunk identifier, and PDF and printed-page ranges.Source: §2.2 · p. 6

2.3 Hybrid Retrieval

• Dense retrieval encodes each evidence chunk and query with a sentence-transformer model and computes semantic relevance using cosine similarity.This represents the semantic component of the retrieval layer.Source: §2.3 · p. 6
• Sparse lexical relevance is estimated using BM25, complementing semantic retrieval with lexical matching.The hybrid formulation is motivated by exact legal phrase matching alongside semantic flexibility.Source: §2.3 · p. 6
• Min–max-normalized semantic and lexical scores are fused, with α controlling the trade-off between them.The stated aim is to support exact legal phrase matching while retaining flexibility for paraphrased queries.Source: §2.3 · p. 6

2.4 Candidate Filtering and Optional Reranking

• Deduplication uses normalized text similarity to reduce repeated evidence from overlapping chunks and improve context diversity before generation.Source: §2.4 · p. 6
• The evaluated heuristic reranker reduced performance in the controlled evaluation, so the final system uses deduplicated hybrid retrieval without reranking.Source: §2.4 · pp. 6–7
• Neural cross-encoder rerankers are excluded because the study focuses on lightweight, locally deployable RAG components.Source: §2.4 · p. 7

2.5 Controlled Evidence-Grounded Generation

• The prompt requires answers to use retrieved sources, cite supporting evidence, avoid external legal knowledge, and refuse when evidence is insufficient.The same grounding, citation, and refusal principles are retained for external datasets.Source: §2.5 · p. 7

2.6 Post-Generation Validation and Regeneration

• Candidate answers are decomposed into sentences and checked for citation validity and grounding before any answer is delivered as final.Source: §2.6 · p. 7
• A sentence is supported only when its citation points to a retrieved source block and it passes either the lexical-overlap threshold of 0.10 or semantic-similarity threshold of 0.40.The thresholds were selected in pilot validation runs and are tunable implementation parameters, not fixed theoretical constants.Source: §2.6 · p. 7
• A response is flagged as hallucinated if it contains any unsupported claim; the supported sentence ratio uses total candidate sentences and unsupported sentences.Source: §2.6 · pp. 8–9
• A grounding-risk signal triggers one stricter-prompt regeneration attempt, and the regenerated candidate is accepted only if compliance improves or insufficient evidence yields a clean refusal.Triggers include invalid or malformed citations, weak support, unsupported content, or refusal despite strong retrieved context; one-pass regeneration limits inference cost and avoids unbounded correction loops.Source: §2.6 · p. 9
• Validation can block unsupported answers but may also reject valid answers when the system behaves conservatively.Source: §2.6 · p. 9

2.7 Controlled Evaluation Set

• The controlled set contains 150 housing-law question–answer pairs from 18 documents across multiple U.S. states and cities, including 123 answerable and 27 unanswerable questions.Gold evidence pages and refusal labels support evaluation of answer generation and abstention.Source: §2.7 · p. 9
• The 150 questions combine 100 human-authored and 50 manually reviewed LLM-assisted formulations, stratified across factual, procedural, reasoning, and edge-case categories.Table 1 reports the question-type distribution.Source: §2.7 · p. 9, Table 1 · p. 9

2.8 External Validation Design

• PrivacyQA provides a fixed 200-question external QA subset about privacy policies, with expert-labeled relevant segments treated as gold evidence.It preserves document-grounded QA structure while shifting from housing law to privacy-policy documents.Source: §2.8 · pp. 9–10
• CUAD is a supplementary 50-question contract-domain stress test, combining legal domain shift with clause-identification and span-like evidence requirements.The stratified subset includes answerable and unanswerable clause-extraction examples.Source: §2.8 · p. 10

2.9 Implementation Details

• The controlled configuration combines normalized semantic and lexical retrieval scores at 0.65/0.35, using 600-token chunks, 120-token overlap, top-k = 8, and up to four retrieved chunks.Dense embeddings use BAAI/bge-base-en-v1.5, and lexical retrieval uses BM25.Source: §2.9 · p. 10
• Generation uses locally deployable instruction-tuned models with temperature 0, and validation failure permits at most one regeneration attempt.The same RAG architecture and validation framework are retained externally, with dataset-specific preprocessing for native evidence structures.Source: §2.9 · p. 10

2.10 Evaluation Protocol

• Evaluation separates retrieval coverage, answer correctness, citation grounding, hallucination control, abstention, and regeneration to distinguish retrieval failures from generation, validation, and refusal-calibration failures.Table 2 summarizes the metric groups.Source: §2.10 · p. 10, Table 2 · p. 11
• Hallucination Rate covers broader answer-level failures, while Unsupported Hallucination Rate specifically measures claims unsupported by retrieved and cited evidence.A benchmark-incorrect response may still be excluded from Unsupported Hallucination Rate if its claims remain grounded in cited evidence.Source: §2.10 · pp. 10–11

3.1 Overall System Performance

• The final selected configuration achieves high retrieval, grounded-answer, citation-validity, and evidence-support performance, with no detected hallucinations and low false refusals.Most candidate answers meet grounding constraints on the first generation pass.Source: Table 3 · p. 11, §3.1 · pp. 11–12

3.2 Component and Configuration Analysis

• BAAI/bge-base-en-v1.5 performs best overall, and hybrid retrieval outperforms lexical-only and semantic-only baselines.The hybrid comparison suggests legal QA benefits from combining exact phrase matching with semantic similarity.Source: §3.2 · p. 12, Table 5 · p. 12, Table 6 · p. 13
• Disabling validation leaves retrieval accuracy unchanged but reduces grounded-answer accuracy and citation validity while increasing hallucination.Citation validity and evidence support therefore cannot be inferred from retrieval success alone.Source: §3.2 · p. 12, Table 7 · p. 13
• The heuristic reranker reduces retrieval and grounded-answer accuracy while increasing false refusals; 600-token chunks and 0.65/0.35 weighting are reported as the preferred settings.The 0.65/0.35 setting has the best overall balance; 0.80/0.20 increases hallucination risk.Source: §3.2 · p. 13, Table 8 · p. 13, Table 9 · p. 13

3.3 External Validation and Stress-Test

• On PrivacyQA, retrieval recall, citation validity, and evidence support remain high with no detected unsupported hallucinations; CUAD retains high recall and citation validity but has lower evidence support and more false refusals.PrivacyQA triggers regeneration for nearly one-third of questions, while CUAD regeneration is less often accepted as the final output.Source: §3.3 · p. 14, Table 11 · p. 14

3.4 Failure Case Analysis

• External errors are more frequent and dataset-specific, including gold-segment mismatch and false answers on PrivacyQA, and over-refusal and clause-span extraction difficulty on CUAD.Residual errors also include false refusals, citation or evidence-compliance failures, segment mismatch, false answers, and incomplete or imprecise evidence selection.Source: §3.4 · pp. 14–15, Table 12 · p. 15

4.1 Findings and Interpretation

• Controlled housing-law answers were source-supported under validation; Mistral and Qwen performed effectively identically, while external transfer retained citation strength but exposed answer and refusal weaknesses.Source: §4.1 · pp. 15–16

4.2 Retrieval and Representation

• Hybrid semantic–lexical retrieval was most reliable because lexical matching preserves exact wording while semantic matching helps with paraphrases, and balanced weighting improves downstream grounding.Excessive semantic weighting increases hallucination risk, while stronger representations improve retrieval and answer reliability.Source: §4.2 · p. 16

4.3 Validation, Grounding, and Regeneration

• Removing validation leaves retrieval accuracy unchanged but sharply decreases grounded-answer accuracy and citation validity; the controlled 0.0% hallucination rate means no unsupported claims were detected against cited evidence.Mistral and Qwen results suggest retrieval, prompting, and validation constraints shape behavior; one-pass regeneration can revise failed candidates but does not fully resolve external over-refusal or span-selection errors.Source: §4.3 · p. 16

4.4 System Design Trade-offs

• Moderate chunking best balances context preservation and irrelevant evidence, while the heuristic reranker suppressed some relevant answer-bearing chunks, leading the final system to omit reranking.The results support optimizing the full retrieval–generation–validation pipeline rather than isolated components.Source: §4.4 · pp. 16–17

4.5 External Validation Insights

• PrivacyQA transfers reasonably in citation control and unsupported-hallucination prevention, whereas CUAD’s clause-extraction task shift increases false refusals; retrieval and citations generalize more robustly than answer extraction.CUAD is a robustness stress test under task shift, not a direct comparison with controlled QA.Source: §4.5 · p. 17

4.6 Limitations and Future Work

• The controlled evaluation uses curated housing law, with external tests limited to PrivacyQA and CUAD; differing answer formats mean accuracy and overlap scores should be read alongside evidence and citation measures.Source: §4.6 · p. 17
• Threshold-based similarity checks may miss implicit entailment, complex legal reasoning, or nuanced paraphrases, and evaluated generators lacked legal-domain adaptation.Regeneration is limited to one attempt; bounded multi-pass recovery may help difficult failures but requires stopping criteria to control inference cost and instability.Source: §4.6 · pp. 17–18

• CiteGuard-RAG combines hybrid semantic–lexical retrieval, citation-constrained generation, sentence-level grounding validation, abstention, and one-pass validation-guided regeneration.Source: §5 · p. 18
• Controlled housing-law results indicate strong retrieval alone is insufficient; post-generation validation is needed to enforce citation validity, evidence support, and hallucination control.Source: §5 · p. 18
• External evaluations on PrivacyQA and CUAD show retrieval coverage and citation validity transfer more robustly than answer extraction and refusal calibration under domain and task shifts.Source: §5 · p. 18