Reducing Hallucinations in Large Language Models Through Integrated Self-Verification and Retrieval-Augmented Generation

1 INTRODUCTION

• Hallucinated engineering material properties or compliance claims can lead to costly mistakes and risk product quality and safety.Engineering uses include CAD documentation, standards compliance assessment, and retrieval of historical design information.Source: §1 · p. 1
• CoVe relies on internal model knowledge, while RAG lacks a mechanism to verify that retrieved information is used accurately.These limitations motivate integrating retrieval with verification.Source: §1 · p. 1
• CoVe-RAG+ combines Chain-of-Verification with Retrieval-Augmented Generation to reduce hallucinations in AI-assisted engineering processes.The framework retrieves external knowledge and checks model outputs against retrieved evidence.Source: §1 · pp. 1–2
• Trials on CAD documentation, compliance validation, and design knowledge retrieval report improved factual correctness, fewer hallucinations, and increased trust in AI outputs.The paper presents the framework as a potential choice for trustworthy engineering design automation.Source: §1 · p. 2

2.1 Hallucination in LLMs

• Hallucinations are factually inaccurate, incomplete, or falsified outputs that can appear confident, and specialized engineering requires current, precise knowledge.Insufficient external grounding and static pretraining are identified as contributing challenges.Source: §2.1 · p. 2

2.2 Chain-of-Verification (CoVe)

• CoVe prompts an LLM to examine its outputs through verification questions, but its reliance on internal knowledge can let it validate incorrect answers in engineering.The approach is described as lowering hallucination rates and improving factual consistency, yet it lacks external data for detailed, updated information.Source: §2.2 · p. 2

2.3 Retrieval-Augmented Generation (RAG)

• RAG grounds answers in external repositories, but it lacks a mechanism to check whether retrieved evidence is interpreted or used accurately.The paper notes that evidence may be altered or improperly combined with internal knowledge.Source: §2.3 · p. 2

2.4 Integrating CoVe and RAG

• CoVe-RAG+ integrates external retrieval with structured verification to support factually reliable engineering outputs, including CAD documentation, compliance checks, and design knowledge reuse.The paper frames the integration as addressing knowledge gaps and reasoning inaccuracies.Source: §2.4 · p. 2

3 METHODOLOGY

• The framework highlights dynamic re-retrieval at low verification confidence, multimodal engineering data, and an explainability layer with source traceability.Supported data include textual standards, CAD data, and simulation design reports.Source: §3 · p. 3

3.1 Overview of CoVe-RAG+

• CoVe-RAG+ combines adaptive retrieval and iterative verification to address knowledge incompleteness and reasoning inconsistencies in compliance-driven engineering workflows.The methodology targets standards compliance validation and CAD documentation.Source: §3 · pp. 2–3, §3.1 · p. 3

3.1.2 Step 2: Adaptive Retrieval

• Adaptive retrieval gathers relevant documents, CAD data, and engineering standards, ranking sources by authority and semantic relevance.The system also considers source legitimacy and recency when ranking documents.Source: §3.1 · p. 3, §3.1.2 · p. 3

3.1.1 Step 1: Task Classification and Query Processing

• The system classifies each query by task type to inform the subsequent retrieval procedure and validation criteria.Examples include compliance verification, CAD documentation, and knowledge retrieval.Source: §3.1.1 · p. 3

3.1.3 Step 3: Baseline Response Generation

• The LLM generates an initial response from retrieved documents, combining direct source references with consolidated insights relevant to the query.Retrieval uses dense semantic search and sparse keyword matching to find relevant documents.Source: §3.1.3 · p. 3, §3.1.2 · p. 3

3.1.4 Step 4: Iterative Chain-of-Verification

• For each detected factual claim, the system generates a verification question, answers it using retrieved documents, and computes a confidence score.Low confidence triggers additional retrieval and re-answering; verification traces record justifications and evidence sources.Source: §3.1.4 · p. 3

3.1.5 Step 5: Final Verified Response Synthesis

• The final response combines the initial answer with validated claims, flags unverifiable claims for user evaluation, and provides confidence scores and supporting proof.The report distinguishes verified and flagged claims.Source: §3.1.5 · p. 3

3.2 Algorithmic Workflow

• Algorithm 1 takes a user query and returns a verified response with an explainability report.It begins by classifying the query and retrieving documents for the task.Source: §3.2 · p. 4
• For each claim in the baseline response, the workflow generates a verification question, answers it using retrieved documents, and calculates a confidence score.If confidence is below the threshold, it retrieves additional documents and re-evaluates the question using updated evidence.Source: §3.2 · p. 4
• The workflow compiles the final response from verified claims and produces an explainability report.Source: §3.2 · p. 4

3.3 Architecture Components

• The six components are a task classifier, multimodal document retriever, LLM generator, verifier, explainability layer, and response synthesizer.Together, they guide retrieval and verification, generate responses and verification questions, record explanations, and integrate verified context into final answers.Source: §3.3 · p. 4

3.4 CoVe-RAG+: Key Innovations and Differentiators

• Multimodal adaptive retrieval brings CAD component information and simulation reports beyond text-based retrieval to improve engineering-context relevance.Retrieval uses varied data types; low-confidence claims can prompt further retrieval and re-verification, while reports provide verifiable proof for final-response claims.Source: §3.4 · p. 4
• Confidence scores trigger dynamic re-retrieval when claims are low-confidence, followed by re-verification.The system also generates user-facing reports with verifiable proof for each factual claim in the final response.Source: §3.4 · p. 4

4 EXPERIMENTAL SETUP

• The evaluation targets correctness, hallucination prevention, and compliance with domain-specific standards in engineering design tasks.The experimental setup frames these as its evaluation objectives.Source: §4 · p. 4

4.1 Datasets

• The Wikidata Engineering Subset contains 45,000 individuals and 80,000 triples; MultiSpanQA (Engineering Edition) pairs engineering questions with multimodal references.The Wikidata subset covers manufacturing entities, standards, and engineering concepts, with expert-checked entries; MultiSpanQA includes references such as CAD specifications and modeling reports.Source: §4.1 · p. 4
• The additional resources include 50,000 GrabCAD CAD files, 15,000 arXiv engineering papers, and 30,000 NIST material datasets.These collections supply CAD models and attributes, engineering papers, and measured or generated material data.Source: §4.1 · p. 4

4.2 Models and Tools

• Mistral 7B and LLaMA 2 65B were fine-tuned on engineering-specific datasets, while LangChain, ElasticSearch, FAISS, and BM25 supported hybrid retrieval.The setup also included modules to parse CAD information and integrate simulation reports.Source: §4.2 · p. 5

4.3 Training Details

• Mistral-7B was fine-tuned with LoRA on 40,000 domain-specific question-answer pairs for five epochs, using batch size 512 and a 5×10−5 learning rate.The dynamic re-retrieval module used LangChain and FAISS for adaptive retrieval and ranking.Source: §4.3 · p. 5

4.4 Evaluation Metrics

• Evaluation combined retrieval precision and recall, FACTSCORE sentence-level factual consistency, and expert checks of truth, readability, and engineering terminology.Domain experts assessed whether generated outputs were factually true, readable, and compliant with engineering terms.Source: §4.4 · p. 5

4.5 Experimental Procedure

• Models answered predefined technical questions, with repeated fact checks and dynamic re-retrieval when confidence thresholds were not met.The experiments ran on dual NVIDIA A100 GPUs, 512GB RAM, and a 64-core AMD EPYC processor.Source: §4.5 · p. 5

5.2 Human Evaluation

• Five mechanical engineering experts reviewed 100 answers for factual correctness, standards compliance, and explanation clarity; inter-rater agreement was strong (Cohen’s Kappa 0.82).Table 2 reports average scores across evaluators, but the supplied passage does not give the score values.Source: §5.2 · p. 5, Table 2 · p. 5

5.3 Statistical Significance Analysis

• Factual-accuracy improvements over Baseline CoVe were statistically significant (p-value < 0.01); Table 1 compares CoVe-RAG+ with baseline CoVe and conventional RAG.CoVe-RAG+ factual correctness had a 95% confidence interval of 4.4 to 4.8.Source: §5.3 · p. 5, Table 1 · p. 5, §5.1 · p. 5

5.4 Error Analysis

• The error analysis attributes 12% of hallucinations to ambiguous queries, 8% of mistakes to outdated standards, and 6% to CAD metadata misinterpretation.The paper identifies real-time standards updates and improved CAD metadata parsing as future improvement opportunities.Source: §5.4 · p. 6

5.5 System Performance and Scalability

• The system uses parallel retrieval and verification, adaptive batching, and confidence-threshold tuning to address delay and computational expense.Threshold changes are intended to reduce unnecessary re-retrievals.Source: §5.5 · p. 6

6 CONCLUSION

• CoVe-RAG+ combines Chain-of-Verification and Retrieval-Augmented Generation in an adaptive, iterative approach to reduce LLM hallucinations.The framework uses dynamic re-retrieval, multimodal data integration, and explainability mechanisms.Source: §6 · p. 6
• The framework achieved a 28% improvement in precision and improved FACTSCORE measures relative to baseline CoVe and RAG methodologies.The conclusion describes the assessments as experimental assessments.Source: §6 · p. 6
• The paper presents CoVe-RAG+ as a scalable, transparent AI solution for fields requiring strong factual integrity.Its stated engineering scope includes standards compliance and CAD documentation.Source: §6 · p. 6

• Multi-stage verification and dynamic retrieval add computation, making CoVe-RAG+ harder to scale and causing delays, especially in real-time applications.Source: §7 · p. 6
• Outdated, missing, or inconsistent external knowledge sources can make verification less accurate.Source: §7 · p. 6
• The framework has only been fully tried in engineering, so its usability and effectiveness in healthcare or law still need testing.Source: §7 · p. 6

• Future retrieval and verification improvements include new indexing approaches, parallelization, and parameter-efficient fine-tuning such as LoRA.Source: §8 · p. 6
• Multimodal capabilities for 3D CAD models, simulation data, and visual media are proposed to extend the framework’s application.Source: §8 · p. 6
• Domain-specific knowledge graphs and ontologies are proposed to advance domain adaptation and generalization in healthcare, aerospace, and legal compliance.Source: §8 · p. 6
• Interactive visuals and simplified reports are proposed to improve user understanding and trust in the explainability layer.Source: §8 · p. 6