PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Specific Hallucination Detection in Large Language Models

Varun Teja Chundru, Debasmita Biswas

arXiv:2609.11878v1cs.CLcs.AIcs.LG

TL;DR

LLM hallucinations create a factuality problem, especially when detectors lack confidence estimates or fail across domains. The paper combines DeBERTa-v3 classification with uncertainty and calibration, applies DPO to generation, and evaluates domain transfer. It reports strong HaluEval detection, reduced detector-scored hallucinations after DPO, and better SciFact adaptation with domain-matched pre-training.

  • Problem

    LLM outputs can contain fabricated or unsupported claims, while detectors may lack confidence estimates and transfer poorly to specialized domains.

  • Method

    The paper combines fine-tuned DeBERTa-v3 detection with MC Dropout, temperature scaling, and ensembles, then applies DPO to a Qwen2.5-0.5B generator and tests SciFact adaptation.

  • Results

    The detector reaches F1=0.915 and AUROC=0.977 on HaluEval, MC Dropout raises accuracy to 93.2%, and DPO lowers detector-scored hallucination from 85.5% to 37.7%.

  • Takeaways & Limitations

    Knowledge context supports detection, DPO reduces measured generator hallucination, and PubMedBERT fine-tuning is the strongest reported SciFact adaptation strategy.

  • Takeaways & Limitations

    The DPO reduction is a co-evaluation because detector and preference supervision share HaluEval-derived supervision, and the detector is not a gold-standard verdict.

Abstract

from arXiv · show

Large language models generate fluent text that can contain unfaithful claims -- a phenomenon known as hallucination. We present a multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo (MC) Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection. Evaluated on the HaluEval benchmark, our pipeline achieves F1=0.915 and AUROC=0.977 on general-domain tasks, with per-task F1 scores of 0.97 (QA), 0.96 (Summarization), and 0.82 (Dialogue). MC Dropout inference further improves accuracy to 93.2%. A context ablation study confirms the model performs genuine entailment reasoning rather than exploiting surface patterns, with summarization F1 dropping 24% when knowledge context is removed. Learning curve analysis reveals that 25% of training data captures 77% of full-data performance. Beyond detection, we apply Direct Preference Optimization (DPO) to a Qwen2.5-0.5B generator, reducing its hallucination rate from 85.5% to 37.7% (55.9% relative reduction) as measured by our detector. Cross-domain evaluation on the SciFact biomedical benchmark shows that general-domain training transfers poorly (F1=0.52), motivating domain-specific fine-tuning. PubMedBERT fine-tuned on SciFact achieves F1=0.63 and AUROC=0.81, demonstrating that domain-matched pre-training is the strongest adaptation strategy. Code and models are available at https://github.com/varunteja99/hallucination-detection-nlp

1 Introduction

The paper frames hallucination detection as an NLI problem and addresses gaps in confidence estimation and specialized-domain transfer with a multi-signal detector and cross-domain evaluation.

  • Hallucination detection is formulated as determining whether a response is faithful to a knowledge source given a prompt and response.
  • Single-model detectors provide point estimates without confidence, while general-domain detectors often fail when specialized terminology and reasoning patterns differ.
  • The paper develops a DeBERTa-v3 pipeline with MC Dropout uncertainty, achieving F1=0.915 and AUROC=0.977 on HaluEval.
  • The study examines detection, mitigation, and cross-domain transfer through context ablation, learning curves, ensemble analysis, DPO, and SciFact evaluation.

2 Related Work

Prior work spans entailment classification, retrieval-based verification, and uncertainty estimation; this paper extends HaluEval analysis with uncertainty quantification and cross-domain evaluation.

  • Hallucination detection research includes entailment-based classification, retrieval-augmented verification, and uncertainty estimation.
  • HaluEval provides balanced QA, dialogue, and summarization samples for controlled task-specific hallucination evaluation.
  • Its analysis extends benchmark evaluation with uncertainty quantification and cross-domain testing.
  • Monte Carlo Dropout estimates epistemic uncertainty through stochastic test-time forward passes and improves accuracy from 91.3% to 93.2%.
  • The paper applies Direct Preference Optimization to hallucination reduction using faithful and hallucinated responses as preference pairs.

3 Methodology

The methodology combines a fine-tuned DeBERTa-v3 NLI classifier with stochastic uncertainty, calibration, and ensemble signals, and uses DPO to train a less-hallucinatory generator.

  • 3.1 Detection Pipeline: The detector uses a shared fine-tuned DeBERTa-v3 backbone with single-pass, MC Dropout, temperature-scaled, and ensemble inference modes.The pipeline includes Simple Average and LR Meta-Classifier combinations.
  • 3.1 Detection Pipeline: MC Dropout averages T=20 stochastic passes to reduce variance and expose epistemic uncertainty on ambiguous inputs.The mean probability serves as the prediction, while standard deviation captures uncertainty.
  • 3.1 Detection Pipeline: Temperature scaling learns a validation-set scalar that changes probability sharpness without changing accuracy or F1.It targets calibration and reduces threshold instability from wrong-but-confident predictions.
  • 3.1 Detection Pipeline: The LR Meta-Classifier combines single-pass probability, MC Dropout mean and variance, and context-response similarity through learned logistic-regression weights.
  • 3.2 DPO Hallucination Mitigation: DPO trains Qwen2.5-0.5B-Instruct on 21K preference pairs, selecting faithful responses over hallucinated ones and evaluating generations with the detector.
  • 3.3 Cross-Domain Transfer: SciFact adaptation compares in-domain DeBERTa fine-tuning, domain-matched PubMedBERT pre-training, and sequential transfer.

4 Experimental Setup

Experiments use stratified HaluEval and SciFact splits, compare detector baselines, and report Accuracy, F1-score, and AUROC for classification and ranking quality.

  • 4.1 Datasets: HaluEval contains 30,000 balanced QA, Dialogue, and Summarization samples, split 70/15/15 into 21,000 training, 4,500 validation, and 4,500 test examples.
  • 4.1 Datasets: SciFact provides 693 labeled claim-evidence pairs extracted from 1,109 claims and 5,183 abstracts, using a stratified 70/15/15 split.
  • 4.2 Baselines: The detector is compared with zero-shot DeBERTa-v3-MNLI and standard single-pass fine-tuned DeBERTa baselines.The baselines isolate training and uncertainty-quantification contributions.
  • 4.2 Baselines: Context ablation is treated as a diagnostic study rather than a competing baseline.
  • 4.3 Metrics: Accuracy measures classification correctness, F1-score combines precision and recall, and AUROC measures ranking quality across thresholds.

5 Experiment 1: Hallucination Detection

On HaluEval, the detection comparison shows MC Dropout as the strongest single-model approach, while performance varies substantially by task.

  • Main Detection Results: 93.2% accuracy and 0.931 F1 make MC Dropout the strongest single-model results, averaging 20 stochastic forward passes without additional training.The comparison evaluates six detection configurations on the HaluEval test set.
  • Main Detection Results: 0.960 AUROC for the LR Meta-Classifier trails MC Dropout's 0.978 despite matching its 0.931 accuracy.The retrieval similarity feature reduces ranking quality because faithful and hallucinated responses share surface vocabulary.
  • Per-Task Analysis: QA reaches F1=0.97, Summarization F1=0.96, and Dialogue F1=0.82, making Dialogue the hardest HaluEval subtask.Dialogue responses are shorter, more implicit, and contain fewer lexical anchors to the knowledge source.

6 Detector Analysis

Diagnostic experiments indicate that the detector uses knowledge context and reaches usable performance with roughly 5K labeled examples, although additional data continues to improve F1.

  • 6.1 Context Ablation: The context ablation removes knowledge source K while retaining [Q; R], directly testing whether detection depends on source comparison.The same fine-tuned weights are reevaluated after context removal.
  • 6.1 Context Ablation: F1 falls from 0.91 to 0.82 without context, with Summarization dropping 24%, QA 1%, and Dialogue from 0.82 to 0.79.The task-dependent degradation supports reliance on the knowledge source rather than response-only surface artifacts.
  • 6.2 Learning Curves: At 25% of training data, 5.3K examples yield F1=0.70, after F1=0.01 at 10% and before 0.82 at 50% and 0.95 at 100%.The learning curve shows a sharp elbow near 5K examples followed by diminishing returns.

7 Experiment 2: Mitigation via DPO

The paper tests whether DPO can reduce generator hallucinations in a detector-in-the-loop setup, finding a large reduction but noting evaluation dependence on shared supervision and the detector’s non-gold-standard status.

  • The closed-loop experiment evaluates 4,500 held-out test generations with the DeBERTa detector scoring both base and DPO outputs.
  • 37.7%: DPO reduces Qwen2.5-0.5B-Instruct’s hallucination rate from 85.5% to 37.7%, a 55.9% relative reduction.The model was trained on 21K preference pairs and evaluated on held-out generations by the DeBERTa detector.
  • The evaluation is co-supervised because the detector and DPO preference pairs both derive from HaluEval.The reported 55.9% reduction is therefore not a fully held-out test.
  • The detector provides a consistent automated signal rather than a gold-standard verdict in this detector-in-the-loop assessment.

8 Experiment 3: Cross-Domain Transfer

General-domain hallucination detection transfers poorly to SciFact, while domain-matched PubMedBERT fine-tuning achieves the strongest cross-domain results among the evaluated configurations.

  • F1=0.517 and AUROC=0.515: zero-shot transfer of the HaluEval-trained DeBERTa detector performs barely above chance on SciFact.The model predicts nearly all scientific claims as hallucinated, reflecting differences in vocabulary, hedging, and citation-grounded reasoning.
  • F1=0.627 and AUROC=0.808: PubMedBERT fine-tuning achieves the strongest SciFact results.Table 4 covers 106 SciFact test examples, and its caption states that Config B wins on every metric.
  • All configurations remain below 0.7 F1 with only 484 SciFact training examples.
  • The reported ranking is domain-matched pre-training > source-task transfer > in-domain training alone > zero-shot.Config C improves AUROC over zero-shot, while Config A improves accuracy but has lower F1 than zero-shot.

9 Conclusion

The paper concludes that its detection pipeline performs strongly on HaluEval, supports hallucination reduction through DPO, and benefits from domain-matched adaptation on SciFact.

  • F1=0.915 on HaluEval and accuracy=93.2% with MC Dropout establish the pipeline’s main reported detection results.
  • Ablations show that removing knowledge context lowers overall F1 by 10 points and summarization F1 by 24%, while approximately 5K labeled examples provide usable performance.
  • DPO reduces generator hallucination rates by 55.9% under the detector.
  • On SciFact, PubMedBERT fine-tuning achieves AUROC=0.808, ahead of source-task transfer and in-domain training alone.
Loading 2609.11878v1…