PaperlayerView exact revision on arXiv

Source-linked AI summary

HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation

Loris Bergeron, Ioana Buhnila, Jérôme François, Radu State

arXiv:2510.00880v1cs.CL

TL;DR

RAG systems can still hallucinate, undermining trust and explainability. HalluGuard combines a small reasoning model with curated synthetic data and preference training to classify claims and provide evidence-grounded justifications. The paper reports competitive performance against larger models, while noting that synthetic claims may not capture all real-world RAG hallucinations.

  • Problem

    RAG remains vulnerable to hallucinations despite retrieved documents, undermining trust and explainability.

    Source: §1 · p. 1

  • Method

    HalluGuard classifies document–claim pairs and supplies evidence-grounded justifications, using curated synthetic data and preference-based training.

    Source: §1 · p. 2, §4.1 · p. 3, §4.3 · p. 5

  • Results

    HalluGuard-4B is reported to rival or surpass much larger LLMs and specialized detectors on LLM-AggreFact, with justifications comparable in quality to those of a 32B-parameter model.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The findings support the conclusion that carefully aligned small reasoning models can deliver reliability and deployability for enterprise RAG applications.

    Source: §8 · p. 8

    Because training relies on synthetic claims, they may not fully capture the nuances of hallucinations in real-world RAG applications.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) excel in many NLP tasks but remain prone to hallucinations, limiting trust in real-world applications. We present HalluGuard, a 4B-parameter Small Reasoning Model (SRM) for mitigating hallucinations in Retrieval-Augmented Generation (RAG). HalluGuard classifies document-claim pairs as grounded or hallucinated and produces evidence-grounded justifications for transparency. Our approach combines (i) a domain-agnostic synthetic dataset derived from FineWeb and refined through multi-stage curation and data reformation, (ii) synthetic grounded and hallucinated claims, and (iii) preference-based fine-tuning with Odds Ratio Preference Optimization to distill large-model reasoning into a smaller backbone. On the RAGTruth subset of the LLM-AggreFact benchmark, HalluGuard achieves 84.0% balanced accuracy (BAcc), rivaling specialized models, MiniCheck (7B; 84.0%) and Granite Guardian 3.3 (8B; 82.2%) while using roughly half their parameters. Over the full benchmark it reaches 75.7% BAcc, matching larger general-purpose LLMs such as GPT-4o (75.9%). We will release HalluGuard and datasets under Apache 2.0 upon acceptance.

1 Introduction

HalluGuard addresses RAG hallucinations with a small reasoning model that classifies document–claim pairs and supplies evidence-grounded justifications. The paper also introduces a synthetic training dataset and reports improved balanced accuracy with competitive performance against larger models.

1 Introduction

  • RAG remains vulnerable to hallucinations despite retrieved documents, undermining trust and explainability.The paper motivates detection and evidence-backed justifications, particularly where users need traceable support from retrieved passages.Source: §1 · pp. 1–2

2 Related Work

  • Prior hallucination detection work mostly uses BERT-based classifiers, which the authors say do not provide justifications when explainability is mandatory.Source: §1 · p. 2

1 Introduction

  • HalluGuard is proposed as a Small Reasoning Model for RAG that classifies claims and generates evidence-grounded justifications.The paper frames these justifications as supporting transparent and reliable use of RAG.Source: §1 · p. 2
  • HalluClaim is a synthetic dataset derived from FineWeb for training and evaluating hallucination detection in RAG-like scenarios.It is described as controlled yet diverse and constructed using Llama3.3-70B.Source: §1 · p. 2
  • The authors report that HalluGuard improves its backbone’s balanced accuracy and performs competitively against larger open-source and closed LLMs.Their ablations highlight reasoning traces, consensus filtering, and ORPO fine-tuning as contributing elements.Source: §1 · p. 2

2 Related Work

  • Related work includes LYNX, an open-source hallucination evaluator with 8B and 70B variants, and Granite Guardian 3.3, an 8B RAG detector with optional reasoning traces.Source: §2 · p. 2

3 Problem Formulation

The task labels document–claim pairs by their relationship, grouping intrinsic and extrinsic hallucinations as hallucinated. HalluGuard is trained to classify these pairs and provide document-grounded justifications through curated synthetic data and preference-based fine-tuning.

3 Problem Formulation

  • Intrinsic hallucinations contradict the document, while extrinsic hallucinations require external knowledge; both are grouped under the hallucinated label.A grounded claim is fully supported by information explicitly present in the document.Source: §3 · p. 2

4.1 HalluGuard Overview

  • HalluGuard predicts whether a document–claim pair is grounded or hallucinated and provides a justification citing the document.At inference, think mode generates intermediate reasoning traces; non-think mode skips them, and the prompt controls the mode.Source: §4.1 · p. 3

4.2 Structured Claim Dataset Construction

  • The training corpus begins with FineWeb and is curated for safety, quality, and diversity before documents are rewritten to broaden linguistic styles.The reformation aims to improve generalization to non-web formats such as reports and dialogues.Source: §4.2 · pp. 3–4
  • One synthetic claim is generated per reformed document, with grounded and hallucinated labels balanced and hallucinations split evenly between intrinsic and extrinsic types.The resulting dataset contains 86,024 balanced document–claim–label triplets.Source: §4.2 · p. 4

4.3 Preference Training Dataset Construction

  • Each preference prompt includes task instructions, a document, and a claim, and requests both a classification and a justification.The larger model's completion is designated chosen and the smaller model's completion rejected to form preference pairs.Source: §4.3 · p. 5
  • Two independent evaluators assess classification correctness, reasoning coherence, and justification clarity; consensus filtering yields 75,360 high-quality preference tuples.The filtering step is described as reducing label noise and mitigating size-based heuristic bias.Source: §4.3 · p. 5

4.4 Preference-Based Fine-Tuning

  • The model uses a Qwen3-4B backbone fine-tuned with ORPO, which aligns chosen and rejected completions while combining supervised fine-tuning and preference alignment without a reference model.LoRA trains small adapter layers while freezing most base weights, reducing memory and compute costs.Source: §4.4 · p. 5

5 Experimental Setup

LLM-AggreFact is a human-annotated benchmark of whether generated claims are supported by evidence documents, spanning diverse domains and including RAGTruth. Performance is measured using balanced accuracy for comparability with prior work.

5 Experimental Setup

  • LLM-AggreFact assesses whether model-generated claims are supported by evidence documents across diverse domains and includes the RAGTruth subset focused on RAG.The benchmark incorporates real hallucinations from recent LLMs.Source: §5 · p. 5
  • Table 1 orders models by average BAcc; HalluGuard-4B and the Qwen3 models were evaluated with the authors’ specific prompt in think mode.Other model results are taken from the public leaderboard.Source: Table 1 · p. 6
  • Balanced accuracy (BAcc) is the evaluation metric, adopted because it was also used in the paper introducing LLM-AggreFact.It is defined using true positives, false negatives, true negatives, and false positives.Source: §5 · p. 6

6 Results

HalluGuard-4B improves on its backbone across LLM-AggreFact and is competitive with larger models. On RAGTruth it matches MiniCheck-7B, while its justifications are comparable to those of a 32B model and its preference construction aligns with human judgments.

6 Results

  • 75.7% average BAcc improves on Qwen3-4B (73.8) by +1.9 points, with gains of +11.2 on WiCE and +8.8 on ClaimVerify.Scores were obtained in think mode with specific prompt and inference parameters.Source: §6 · p. 6
  • HalluGuard-4B scores 75.7% BAcc, near GPT-4o (75.9), while trailing Granite Guardian 3.3 (76.5) by 0.8 points and MiniCheck-7B (77.4) by 1.7.The specialized baselines have 8B and 7B parameters, respectively; HalluGuard-4B surpasses them on some benchmarks.Source: §6 · p. 6
  • 84.0% average BAcc on RAGTruth matches MiniCheck-7B (84.0) and exceeds Granite Guardian 3.3 (82.2) using roughly half their parameters.HalluGuard-4B has a TPR of 77.7% and TNR of 90.7%; Table 2 presents its confusion matrix.Source: Table 2 · p. 6, §6 · p. 6
  • HalluGuard-4B achieves justification quality comparable to Qwen3-32B despite being almost an order of magnitude smaller, while fluency remains uniformly high.The evaluation covers relevance, consistency, coherence, and fluency.Source: Table 3 · p. 7, §6 · p. 7
  • 94.7% of 75 fully agreed items and 83.5% of 200 individual judgments favored the chosen completions in the human alignment evaluation.Annotators evaluated correctness, reasoning coherence, and justification clarity while blind to the constructed preference labels and presented with randomized completion order.Source: Table 4 · p. 7, §6 · p. 7

7 Ablation Study

The full HalluGuard-4B pipeline outperforms its ablated variants. Ablations indicate contributions from consensus filtering, reasoning traces, and preference alignment, with the largest reported drop when ORPO is replaced by SFT alone.

7 Ablation Study

  • 75.7% BAcc with consensus filtering compares with 75.3% without it, a +0.4-point improvement; without filtering, HalluGuard falls below Qwen2.5-72B-Instruct (75.6%).Filtering uses independent evaluators IE-1 and IE-2.Source: §7 · p. 8
  • 75.7% BAcc in think mode drops to 67.6% with /no_think (−8.1 points); reasoning improves RAGTruth BAcc by +21.8%.The reasoning gain is consistent across the other datasets as shown in Figure 5.Source: §7 · p. 8, Figure 5 · p. 8
  • 75.7% BAcc with ORPO falls to 48.1% with SFT alone, a −27.6-point change and the largest drop in the ablation study.The authors identify preference alignment as important to reasoning reliability and quality.Source: §7 · p. 8
  • The full HalluGuard-4B model consistently outperforms all three ablated variants on LLM-AggreFact.The results indicate that robustness arises from component interaction rather than any single factor.Source: §7 · p. 8

8 Conclusion

HalluGuard is a 4B-parameter Small Reasoning Model for RAG that detects hallucinations and provides evidence-grounded justifications. The conclusion reports competitive benchmark performance and justification quality comparable to a 32B model.

8 Conclusion

  • HalluGuard combines a domain-agnostic synthetic dataset with multi-stage curation and preference-based fine-tuning via ORPO and LoRA.The authors report that it rivals or surpasses much larger LLMs and recent specialized hallucination-detection models.Source: §8 · p. 8
  • HalluGuard-4B achieves competitive performance on LLM-AggreFact while providing justifications comparable in quality to those of a 32B-parameter model.Source: §8 · p. 8
  • Ablations highlight reasoning traces, consensus filtering, and preference alignment as important to the reported gains.Source: §8 · p. 8
  • HalluGuard and its datasets are slated for release under Apache 2.0 upon acceptance.Source: §8 · p. 8

9 Future Work

Future work targets finer hallucination distinctions, multimodal support, and larger Qwen3-based variants.

9 Future Work

  • Future work will distinguish intrinsic from extrinsic hallucinations, investigate multimodal extensions for charts, and release 8B and 14B Qwen3-based variants.Source: §9 · p. 8

Limitations

The paper identifies limits in synthetic-data coverage, output-format scoring, hallucination labels, and language and domain generalization. It also recommends human oversight rather than fully autonomous use.

Limitations

  • Synthetic claims may not capture all nuances of hallucinations in real-world RAG applications.Source: Limitations · p. 9
  • Strict JSON output requirements can cause performance to be underestimated when deviations are scored as incorrect.Source: Limitations · p. 9
  • Merging intrinsic and extrinsic hallucinations into one label reduces explainability when distinguishing those types matters.Source: Limitations · p. 9
  • HalluGuard was trained and evaluated on English data, leaving performance in other languages and specialized domains uncertain.Source: Limitations · p. 9

Ethical Considerations

  • The authors recommend decision-support use with human oversight because overflagging or missed hallucinations may reduce trust or lead to downstream harm.Source: Ethical Considerations · p. 9

A Technical Reproducibility

The supplied reproducibility and appendix passages report the experimental environment and illustrate data-reformation and synthetic-pair prompts. They specify grounded and two hallucinated claim-generation instructions, followed by a classification-and-justification task.

A Technical Reproducibility

  • Training used one NVIDIA H100 PCIe GPU with 80GB memory for 16 hours and consumed approximately 7.35 kWh.Experiments used Linux, CUDA 12.4.1, PyTorch 2.4.0, and default random seeds and settings.Source: A Technical Reproducibility · p. 12

B Prompt Template: Style Reformation

  • The style-reformation template applies each style to reform FineWeb raw data and increase stylistic diversity.Source: Table 5 · p. 12

grounded

  • The grounded-claim prompt asks for a factually accurate claim fully supported by context, with no added explanation or formatting.Source: grounded · p. 12

hallucinated_intrinsic

hallucinated_extrinsic

  • The extrinsic-hallucination prompt asks for a plausible claim that cannot be verified within the context and requires external knowledge.Source: hallucinated_extrinsic · p. 12

D Prompt Template: Synthetic Pairs

  • The synthetic-pair evaluation prompt classifies document–claim pairs as GROUNDED or HALLUCINATED and requires a justification.The latter label includes claims that contradict the document or cannot be verified from it.Source: D Prompt Template: Synthetic Pairs · pp. 12–13

E Prompt Template: Consensus Filter {

The appendix presents a consensus filter for choosing between responses, surveys LLM-AggreFact datasets, and reports training reward trends and configuration details.

E Prompt Template: Consensus Filter {

  • The consensus filter selects the better of two responses based on classification correctness, thinking coherence and clarity, and justification quality.Its prompt requires the selection as RESPONSE_A or RESPONSE_B without justification.Source: E Prompt Template: Consensus Filter { · p. 13

F Benchmark Datasets

  • LLM-AggreFact spans summarization, dialogue, entailment, attributed open-domain QA, search fact-checking, expert queries, long-form answers, and retrieved-passage RAG.RAGTruth annotates outputs grounded in retrieved passages.Source: F Benchmark Datasets · p. 13

G Reward Gap Across Training Epochs

  • The chosen–rejected response gap increases over training, indicating growing preference for chosen examples and lower rewards for rejected ones.Figure 6 depicts this trend across training epochs.Source: Figure 6 · p. 13

H Inference Parameters

  • Inference settings follow recommended Qwen parameters for non-thinking and thinking modes.Table 6 presents the inference parameters used in the experiments.Source: Table 6 · p. 13

I Fine-Tuning Configuration

  • LoRA fine-tuning trains approximately 33M parameters, 0.81% of the full model, for one epoch.These are the training settings summarized in Table 7.Source: Table 7 · p. 14

J ORPO Preference Tuple: Full Example

The example prompt classifies a document–claim pair as grounded or hallucinated and provides a justification; it also shows response candidates organized as a preference pair.

J ORPO Preference Tuple: Full Example

  • The example requests a classification and justification for a document–claim pair.Its response fields are CLASSIFICATION and JUSTIFICATION.Source: J ORPO Preference Tuple: Full Example · p. 14
  • A grounded claim is directly verifiable and fully supported by the document, while a hallucinated claim contradicts it or includes unverifiable information.The prompt labels these cases GROUNDED and HALLUCINATED.Source: J ORPO Preference Tuple: Full Example · p. 14
  • The example includes assistant responses labeled chosen and rejected, with one response classifying the claim as hallucinated and another as grounded.The shown justifications respectively describe contradiction and direct document support.Source: J ORPO Preference Tuple: Full Example · p. 14
Loading 2510.00880v1…