The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

Introduction

• Because common automated label sources are often used without direct validation against the intended construct, their labels may affect benchmark evaluation outcomes.The paper focuses on open-domain QA benchmarks targeting factual correctness, not on evidence-grounded tasks where faithfulness may be intended.Source: §1 · p. 2
• Reference faithfulness treats any claim unsupported by the reference as hallucinated, whereas factual correctness requires a contradiction or an untrue specific claim.The criteria diverge when a correct answer adds true information absent from the reference.Source: §1 · p. 2
• The evaluation compares lexical metrics, reference-entailment NLI, and seven LLM judges with newly collected human annotations and controlled prompt variants.It also examines agreement and false-positive and false-negative error structures.Source: §1 · p. 2
• Across a 900-pair audit, faithfulness-style labelers were false-positive-dominant, while factual-correctness prompts reduced this bias across most judges.The study also found that judge capability alone did not determine the size of faithfulness-induced bias.Source: §1 · p. 2

2 Related Work

• Benchmarks vary between factual correctness and faithfulness to provided evidence, so results across tasks may not be directly comparable without accounting for the construct measured [Li et al., 2023, Maynez et al., 2020, Niu et al., 2024].TRUE [Honovich et al., 2022] standardizes factual-consistency datasets from diverse tasks under a common evaluation protocol.Source: §2 · p. 2
• The evaluated detectors generally do not take benchmark reference answers as inputs; those answers are instead used separately to construct evaluation labels [Farquhar et al., 2024, Duan et al., 2024, Qiu and Miikkulainen 2024, Chen et al. 2024, Yin et al. 2024].Detector methods may use generated-sample uncertainty, semantic density, or internal representations.Source: §2 · p. 3
• QA label rules often compare answers with references, while some sentence-length GPT-4 judgments assess meaning without separately verifying factual correctness [Duan et al. 2024; Qiu and Miikkulainen 2024; Chen et al. 2024; Yin et al. 2024; Farquhar et al. 2024].The authors characterize these strategies as operationalizing reference faithfulness, with a paragraph-length FACTUALBIO evaluation as an exception.Source: §2 · p. 3
• Claim-level verification makes factual assessment more fine-grained, but does not by itself resolve whether benchmark hallucination labels target reference faithfulness or factual correctness [Min et al., 2023, Hu et al., 2024].These approaches clarify the unit of evaluation, not the criterion used to define hallucination.Source: §2 · p. 3
• Automatic evaluation signals can diverge from human judgments, and QA meta-evaluations report that misaligned signals can overstate progress or misrank methods [Maynez et al., 2020, Honovich et al., 2022, Pagnoni et al., 2021, Janiak et al., 2025, Kulkarni et al., 2025].This work therefore examines whether labels used to evaluate detectors reflect the intended construct.Source: §2 · p. 3
• LLM-judge reliability varies with capability and prompt design, and this study asks whether criterion mismatch contributes to directional false-positive errors in short-reference QA [Zheng et al., 2023, Thakur et al., 2025].The focus is construct-level misalignment between a reference-faithfulness label source and a factual-correctness target.Source: §2 · p. 3

3.1 Labeling Constructs: Faithfulness vs. Factual Correctness

• The faithfulness criterion flags any claim unsupported by the reference, while factual correctness flags contradictions or untrue factual claims; unsupported does not necessarily mean contradicted.A correct answer with a true elaboration absent from the short reference is a case where the criteria diverge.Source: §3.1 · p. 4

3.2 Dataset Generation

• Questions came from TriviaQA, HotpotQA, and TruthfulQA, representing factoid, multi-hop, and misconception-eliciting QA settings; each selected question received answers from three open-weight generators.The generators were Llama-3-8B, Gemma-2-9B, and Mistral-7B, with the same question IDs used across models for paired comparisons.Source: §3.2 · p. 4

3.3 Human-Annotated Benchmark

• The audit contains 900 pairs from 300 sampled questions, with all three generator answers retained; independent annotators achieved Cohen’s κ = 0.808 on a 300-pair subset.Downstream evaluations used the primary annotator’s labels; the audit is diagnostic, not a population-level estimate of hallucination prevalence.Source: §3.3 · pp. 4–5

3.4 Automated Label Sources

• Label sources included ROUGE-L and BERTScore thresholds, strict reference-entailment NLI, and seven LLM judges; GPT-5-mini was tested under faithfulness and factual-correctness prompt variants.The lexical thresholds included oracle and cross-validated protocols; the strict NLI and similarity metrics operationalize reference faithfulness.Source: §3.4 · p. 5

3.5 Evaluation Protocol

• Automated labels were compared with human factual-correctness labels using Cohen’s κ, positive-label bias, false-positive and false-negative counts, and error rates.Paired GPT-5-mini prompt comparisons used κ differences, bootstrap confidence intervals, and McNemar’s test on the same question–answer pairs.Source: §3.5 · pp. 5–6
• The remaining judges tested whether the p1-to-p2 pattern generalized across model families and capabilities, while pairwise agreement matrices and error-direction plots compared label-source structure.For multi-prompt judges, the plots tracked movement from p1 to p2 to p3.Source: §3.5 · p. 6

4.1 Prompt Criterion Changes Label Behavior

• A factorial control with GPT-5-mini attributes the p1→p2 agreement gain to criterion choice rather than prompt elaboration.Matching p1's structure to p2 yields no detectable improvement, and condensing p2 to p1's structure does not reduce agreement.Source: §4.1 · p. 6
• Replacing p1 with factual-correctness p2 generally improves human-label agreement and reduces false-positive dominance; p3 adds no consistent gain over p2.For GPT-5-mini, p2 is significantly more often correct than p1 for all three generators (p < 0.0001), while no p2→p3 difference is statistically significant for any generator.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Lexical and NLI Baselines Fail in Different Directions

• ROUGE-L is false-negative-prone, BERTScore varies across generators, and strict NLI is almost entirely false-positive-dominant (FP%: 98.3–99.4%).Oracle thresholding improves some baseline scores but does not bring them into the agreement range of criterion-aligned API judges.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Error Direction Separates Label Sources

• API judges generally move toward fewer false positives under p2, while local judges vary and Llama-3-8B shifts toward false-negative dominance under p2/p3.Among non-LLM baselines, strict NLI is false-positive-dominant, ROUGE-L is mostly false-negative-prone, and BERTScore is mixed.Source: §4.3 · pp. 7–8, Figure 1 · p. 8

5.1 Faithfulness Bias as Measurement Error

• Removing elaborative material reduced the false-positive rate from 52.9% to 23.2% (45 FP→TN vs. 4 TN→FP transitions), supporting a directional faithfulness bias.The reported false positives are relative to the factual-correctness target.Source: §5.1 · p. 9

5.2 The Main Intervention Is Criterion Alignment

• Changing the criterion improves agreement at either prompt-structure level, whereas added decision structure alone produces no detectable change.The key distinction is that absence from a short reference is not itself contradiction.Source: §5.2 · p. 9

5.3 Judge Capability Is Not Enough

• Frontier judges differ under nominal p1 but move toward greater similarity under p2; local judges show lower agreement and model-specific robustness limits.The experiments do not identify the mechanism behind the cross-model differences.Source: §5.3 · p. 9
• Lexical similarity and strict NLI impose distinct criteria, potentially penalizing factually correct information absent from the reference.Similarity can penalize elaboration, while strict entailment can treat unreferenced information as non-entailed.Source: §5.3 · p. 9

5.4 Implications and Limitations

• Benchmarks should specify the target criterion, validate label sources against it, and interpret reference-similarity results only under that operational rule.For factual correctness, prompts should distinguish contradiction from absence and allow correct elaboration.Source: §5.4 · pp. 9–10
• Evidence is bounded to English short-reference QA, three small open-weight generators, A1 labels, and fixed generation and judging setups.The authors call for larger samples and validation across further task families, longer answers, domains, and generators; binary labels also cannot capture severity or partial correctness.Source: §5.4 · p. 10

• Faithfulness-style labelers may penalize factually correct additions absent from short references, creating measurement error when factual correctness is the target.Source: §6 · p. 10
• Across seven LLM judges, three generator families, lexical metrics, and reference-entailment NLI, label-source choice changes human agreement and error direction.Faithfulness-style prompts are often false-positive-dominant, while factual-correctness prompts reduce this bias for most judges; local judges also show robustness limits.Source: §6 · p. 10
• Benchmarks should state whether they target reference faithfulness or factual correctness and validate automated label sources against that intended criterion.For short-reference QA, absence from the reference should not count as hallucination unless contradicted or factually false.Source: §6 · p. 10

A Annotation Protocol and Agreement

• The answer-level factual-correctness label is 1 for a reference contradiction or a specific verifiable false claim, including one resolved through external adjudication.The reference is primary evidence, but its silence alone does not make an answer hallucinated; unsupported but true elaborations are not penalized.Source: A Annotation Protocol and Agreement · p. 14
• Vague, incomplete, or non-committal answers are not hallucinations unless they contradict the reference or contain a specific, verifiable false claim.This separates hallucination labels from general answer quality, and allows correct information absent from the short reference.Source: A Annotation Protocol and Agreement · p. 14
• κ = 0.808 (95% CI: [0.729, 0.879]) on the 300-item overlap subset, with 90.3% raw agreement (271/300).Dataset estimates were highest for HotpotQA (κ = 0.832) and lower for TruthfulQA (κ = 0.690), but their intervals are wide and differences are descriptive.Source: Table 3 · p. 15, A Annotation Protocol and Agreement · p. 15
• A1 labeled all 900 benchmark items, while A2 independently labeled a 300-item subset for reliability checks without replacing A1 labels with consensus.The authors caution that the reliability analysis does not establish that benchmark labels are error-free.Source: A Annotation Protocol and Agreement · p. 15

B Judge Prompts

• The prompts distinguish a reference-faithfulness baseline (p1) from factual-correctness prompts (p2 and p3), with p3 adding non-committal-reference handling.p1 labels unsupported or contradicted claims; p2 requires contradiction or a specific false detail, while p3 adds an explicit branch for non-committal references.Source: B Judge Prompts · pp. 15–17

C.1 Answer Generation Setup

• All three generators used the same instruction-tuned chat-template setup and decoding parameters: temperature 0.7, top-p 0.95, 128 maximum new tokens, and seed 42.The generators were Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-IT.Source: §C.1 · p. 18

C.2 Judge Evaluation Setup

• Judge outputs must be strict JSON with a binary label and numeric confidence between 0 and 1; unparseable responses are excluded only when automatic repair fails.Table 4 summarizes judge identifiers, providers, and experimental roles.Source: §C.2 · p. 18, Table 4 · p. 18
• Table 4 lists judge model identifiers and evaluation roles, with API access dates reported in the corresponding bibliography entries.The table caption does not specify individual judge rows or their values.Source: Table 4 · p. 18

C.3 NLI Fine-tuning Datasets

• The reference-entailment baseline is a fine-tuned T5-11B model using SNLI, MNLI, Fever, SciTail, PAWS, and VitaminC datasets.The authors report conclusions specific to this T5-11B instantiation and do not claim it is optimal or representative of NLI systems generally.Source: §C.3 · p. 18

D.1 Six-Branch Sanity Check

• All seven judges pass the six-branch p2 diagnostic, including the case where an answer adds true information absent from the reference.The authors interpret this as evidence that the prompt is understandable in simple controlled cases, not as benchmark-performance evidence.Source: §D.1 · pp. 18–19, Table 5 · p. 19

D.2.1 Paired Prompt Comparisons

• The paired comparison uses Cohen’s κ for chance-corrected agreement and an exact two-sided McNemar test for item-level correctness on the same generated answers.Bootstrap intervals quantify sampling uncertainty in Δκ.Source: §D.2.1 · p. 19
• Switching from p1 to p2 yields large, statistically significant human-agreement gains across generators; the largest is Mistral-7B at Δκ = +0.432.p1-to-p3 gains are similar, while p2-to-p3 differences are not significant; this does not establish exact equivalence or a consistent benefit from p3’s extra structure.Source: §D.2.1 · p. 19, §D.2 · p. 19

D.2.2 Per-Dataset Breakdown

• Switching from p1 to p2 increases agreement with human labels across all three datasets and all generators.Gains are largest on TriviaQA and TruthfulQA and smaller on HotpotQA, except for Mistral-7B, whose HotpotQA gain is also large.Source: §D.2.2 · p. 19

D.3 Labeling Criterion vs. Prompt Structure Factorial Ablation

• GPT-5-mini tested four criterion-by-format combinations on the same 900 responses in one interleaved run, including structured faithfulness p1s and terse factual-correctness p2t.The auxiliary prompts paired each criterion with the opposite format used by its main prompt.Source: §D.3 · pp. 19–21
• The prompts distinguish faithfulness, which labels unsupported claims as hallucinations, from factual correctness, which permits true information absent from the reference.The factual-correctness definition treats contradicted claims or specific false details as hallucinations, but not true absent information.Source: §D.3 · pp. 20–21
• Both factual-correctness prompts achieve higher human-label agreement than either faithfulness prompt; changing criterion improves agreement in both formats (Table 8; Table 9).The criterion contrasts’ confidence intervals exclude zero, while the terse-versus-structured difference under factual correctness is small and its interval includes zero.Source: §D.3 · p. 21, Table 8 · p. 22, Table 9 · p. 22
• The factorial ablation suggests criterion matters more than added structure, while structure’s effect depends on whether the prompt uses faithfulness or factual correctness.Under faithfulness, structured p1s has lower agreement and more false positives than terse p1.Source: §D.3 · p. 21, Table 9 · p. 22

D.4 METEOR and BARTScore-CNN Baselines

• κ was 0.405 for METEOR, 0.335 for BARTScore-CNN ref→hyp, and 0.411 for BARTScore-CNN hyp→ref, within the main-text baseline range.The comparison baselines were ROUGE-L at 0.433, BERTScore at 0.429, and threshold-tuned NLI at 0.330.Source: §D.4 · p. 22, Table 10 · p. 23

D.5 Full Prompt-Ablation Results

• The full prompt-ablation results are provided in an audit table listing each judge–generator–prompt combination with bias and false-positive/false-negative error decomposition.The table marks the best κ within each judge–generator block.Source: §D.5 · p. 23

D.6 Per-Generator Inter-Method Agreement Matrices

• API factual-correctness judges form a coherent block, with within-block agreement from κ = 0.719 to κ = 0.879 (mean κ = 0.781).Local p2 judges align less tightly, with κ = 0.531–0.660.Source: §D.6 · p. 23
• Across all three generator matrices, criterion-aligned factual-correctness judges group together, while faithfulness-style sources show heterogeneous alignment.NLI strict and GPT-5.4 p1 remain weakly aligned; Claude Opus 4.7 p1 is much closer to the factual-correctness judges.Source: §D.6 · p. 23
• GPT-5-mini p1 is intermediate, whereas Claude Opus 4.7 p1 already agrees strongly with factual-correctness judges, breaking the usual p1-to-p2 pattern.The matrices support criterion-level organization, modulated by judge capability and individual behavior.Source: §D.6 · p. 23

E Paired elaboration-removal intervention

• The intervention used deletion-only core extraction on 140 answers both humans labeled non-hallucinated, preserving answer-bearing content without adding, rewriting, or reordering words.Two of 140 extractions failed blinded human validation, leaving 138 paired cases; mean answer length fell from 48.4 to 15.6 words.Source: E Paired elaboration-removal… · pp. 23, 25
• False positives fell from 52.9% (73/138) on original answers to 23.2% (32/138) on core-only answers, a 29.7-point absolute reduction.The 95% paired bootstrap confidence interval was 21.0–38.4 percentage points; the relative reduction was 56.2%.Source: E Paired elaboration-removal… · p. 26
• The item-level changes were asymmetric: 45 answers shifted from false positive to true negative, versus 4 in the opposite direction (exact McNemar’s test, p = 8.23 × 10^-10).Elaboration contributed to p1 false positives, but its removal did not eliminate them.Source: E Paired elaboration-removal… · p. 26
• The remaining text in the examples was generic elaboration—such as hedging, procedural caveats, or context absent from the short reference—around a correct core answer.A faithfulness-style prompt cannot readily distinguish these elaborations from unsupported factual assertions because the reference does not entail either.Source: E Paired elaboration-removal… · p. 26