PaperlayerView exact revision on arXiv

Source-linked AI summary

Decision Models as Judges for Hallucination Detection

Haozheng Liu, Zhaoqian Xue, Zhongyin Wang, Junting Mu, Zihui Ma, Shuya Feng, Lingyao Li

arXiv:2610.37229v1cs.CL

TL;DR

This paper asks whether decision models can provide reliable, lower-cost judgments for hallucination detection across error types. It evaluates five decision models and ensembles against gpt-6-sol, finding type-dependent performance and confidence reliability, with cascades matching or exceeding the comparator on some types.

  • Problem

    Evidence remains limited on decision-model accuracy across hallucination types, confidence reliability across models and types, and whether decision-model-only ensembles improve detection at low cost.

    Source: §2 · p. 2

  • Method

    The authors evaluate five decision models against gpt-6-sol on hallucination benchmarks and test confidence-based cascades of decision models.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Decision models match or approach gpt-6-sol on factual and context-related errors but lag on logical and instruction inconsistencies; cascades match or exceed it on three types.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Judge choice and confidence-based deferral should be validated for the target hallucination type, since model performance and cascade gains vary by type.

    Source: §5 · p. 8

    Coverage is uneven, with context and instruction inconsistency each relying on one source, while data-construction and domain differences confound cross-dataset comparisons.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models are widely used as judges of model outputs (LLM-as-a-Judge), but they are costly and can hallucinate or be overconfident. Decision models are a potential alternative. They return a predefined verdict with a probability for each label instead of free-form text and are much cheaper and faster. We test this promise on hallucination detection. We build binary benchmarks for five hallucination types (9,852 items from six datasets) and compare five decision models with the generative judge gpt-6-sol. Performance depends strongly on the type: the best decision model matches gpt-6-sol in balanced accuracy on context inconsistency and comes within 3 percentage points on factual errors, but trails it by 4-8 points on logical and instruction inconsistency. Decision-model probabilities are overconfident, though less so than the GPT-6 models, and separate correct from incorrect decisions only moderately. We also find that a confidence cascade of decision models improves on the best single model by 1.9-3.3 points on three types and matches or exceeds gpt-6-sol at 1-3% of its cost. Overall, decision models are a practical low-cost judge for some hallucination types but need to be validated before use.

1 Introduction

LLMs are widely used to judge model outputs, but their judgments can be unsupported or overconfident. This study tests decision models as lower-cost judges for hallucination detection, examining accuracy across error types, confidence reliability, and ensembles.

1 Introduction

  • LLM judges can produce incorrect or unsupported judgments and be overconfident.Source: §1 · p. 1
  • Decision models return predefined verdicts with probabilities and are presented as cheaper and faster alternatives to generative LLM judges.Their probabilities could indicate when a judgment should not be trusted if reliable.Source: §1 · p. 1
  • Prior decision-model judge studies examine single models, leaving cross-type performance, confidence reliability, and decision-model-only hallucination ensembles underexplored.Source: §1 · p. 1
  • The study evaluates five decision models against gpt-6-sol on 9,852 samples from six datasets spanning five hallucination types.It reports type-dependent performance, confidence overconfidence, and accuracy improvements from confidence cascades.Source: §1 · p. 2

2 Related Work

Prior work establishes generative judges, hallucination benchmarks, confidence measures, and judge ensembles, while leaving decision-model hallucination detection and decision-model-only ensembles insufficiently examined.

2 Related Work

  • Generative judges are used for evaluation but can show position and verbosity biases, while decision models select predefined labels and return probabilities.Prior work reports lower cost and latency for decision models than generative judges.Source: §2 · p. 2
  • Hallucination benchmarks cover factual errors, retrieval-grounded support and contradiction, document or reasoning contradictions, and instruction following.AggreFact reports that factuality-detector performance varies across datasets, summarizers, and error types.Source: §2 · p. 2
  • Confidence calibration and error ranking are distinct reliability questions, and decision-model calibration varies by workload.Prior work also reports that high confidence can coexist with near-chance accuracy.Source: §2 · p. 2
  • Prior judge ensembles aggregate diverse generative models or use disagreement and confidence-based cascades, with overlapping errors limiting some gains.Source: §2 · p. 2
  • Prior decision-model judge studies assess individual models or tasks other than hallucination detection, and leave cross-type accuracy, confidence reliability, and decision-model-only ensembles open.Source: §2 · p. 2

3 Data and Methods

The methods construct five separate binary hallucination tasks from six labeled datasets, evaluate five decision models and a generative comparator, measure accuracy and confidence, and test three ensemble strategies.

3.1 Benchmark Composition

  • Five hallucination types are treated as separate binary tasks, and six datasets with explicit target-error annotations supply the benchmark items.A negative label means absence of the target error, not necessarily absence of other hallucination types; class balance is unchanged.Source: §3.1 · pp. 2–3

3.2 Evaluation Setup and Metrics

  • Five decision models and gpt-6-sol receive the same type-specific criterion and response, with instruction and, for context inconsistency, source context when available.Models return H/N probabilities; the higher-probability label is the answer, with the model's own answer retained on an exact tie. Evaluation reports balanced accuracy, macro-F1, latency, and estimated API cost.Source: §3.2 · pp. 3–4

3.3 Confidence Analysis

  • Confidence analysis measures detection, calibration, and whether confidence ranks correct decisions above errors.Calibration uses expected calibration error over ten equal-width bins and a logistic calibration slope, where a slope below 1 indicates overconfidence; confidence is the probability of the predicted label.Source: §3.3 · p. 4

3.4 Model Ensembling Strategies

  • Three ensemble methods combine the same top three decision models per type: majority voting, arbitration, and confidence cascades.Cascades defer least-confident items at target deferral rates; the best rates are selected from 36 combinations on the evaluation items, with five-fold cross-fitting also used to assess optimism.Source: §3.4 · pp. 4–5

4 Results

Across hallucination types, judge performance and confidence reliability vary substantially. Confidence cascades improve on single judges for selected types, but their gains do not transfer uniformly.

4.1 RQ1: How Do Decision Models Perform Across Hallucination Types?

  • Pooled BAcc varies by type: gpt-6-sol leads FC, LI, and II; gpt-6-luna scores 82.7 vs 82.0 on FF, while three judges cluster near 82.6 on CI.Type and source dataset are confounded, so the reported differences should not be attributed to hallucination type alone.Source: Table 2 · p. 5, §4.1 · p. 5, Figure 2 · p. 6
  • Similar CI BAcc hides opposite errors: pplx has a 31.5% false-negative rate, while gpt-6-sol has a 28.8% false-positive rate.BAcc alone does not show whether a judge mainly misses hallucinations or raises false alarms.Source: §4.1 · pp. 5–6
  • On HADTest, all six judges score 12–18 points higher on FF than FC, while FAVABench shows no consistent gap; CI and II each have one source.Thus, observed advantages can depend on source, and cross-source stability remains untested for CI and II.Source: §4.1 · p. 6

4.2 RQ2: How Reliable is Their Confidence?

  • Decision models are overconfident, but less so than GPT-6 models: their mean confidence exceeds accuracy by 0.04–0.12, versus gaps of 0.18 for gpt-6-luna and 0.16 for gpt-6-sol.Dataset accounts for 58% of overconfidence-gap variance across decision models, compared with 15% for model.Source: §4.2 · p. 6

Context inconsistency (CI)

  • Confidence reliability varies by type: II has the largest decision-model overconfidence gap (0.14) and lowest conf-AUROC (0.64), while CI and FF have the highest conf-AUROC (0.74 and 0.75).On CI, gpt-6-luna accuracy stays below 0.50 at confidence levels from 0.55 to 0.95.Source: Context inconsistency (CI) · p. 7

4.2 RQ2: How Reliable is Their Confidence?

  • Confidence separates correct from incorrect decisions only moderately (conf-AUROC 0.67–0.73), and 10–30% of decision-model errors occur at c ≥0.9.Across models and types, confidence intervals for conf-AUROC lie above 0.5, indicating some information about correctness.Source: Context inconsistency (CI) · p. 7

4.3 RQ3: Can Ensembling Improve the Performance?

  • On FC, FF, and CI, the cascade scores 82.2, 84.6, and 84.6 BAcc—0.6, 2.6, and 1.9 points above gpt-6-sol—at 1.2–3.3% of its estimated cost.It improves over the strongest single member by 3.3, 1.9, and 2.0 points, respectively; cross-fitting reduces BAcc by at most 0.52 points while preserving those gains.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
  • The cascade beats voting and arbitration across types, but its best settings vary: it gives no LI gain and remains 7.1 points below gpt-6-sol on II.FC, FF, and CI peak at first-stage deferral rates of 20–30%; a single cascade configuration does not transfer across types.Source: §4.3 · p. 8

5 Discussion and Conclusion

Decision models match or approach the generative judge on factual and context-related errors but lag on logical and instruction inconsistencies. Their usefulness therefore depends on error type, confidence limits, and whether cascade gains justify additional calls.

5 Discussion and Conclusion

  • Decision models match or approach gpt-6-sol on factual and context-related errors but lag on logical and instruction inconsistencies requiring multi-step reasoning or explicit constraint checking.The discussion recommends reserving LLM judges for logical and instruction inconsistencies.Source: §5 · p. 8
  • No decision model performs best across hallucination types, and similar balanced accuracy can hide different profiles of missed hallucinations and false alarms.Judge selection should reflect which error is more costly and be validated on the target task because type and source are partly confounded.Source: §5 · p. 8
  • Decision-model confidence only moderately distinguishes correct from incorrect judgments, with frequent high-confidence errors on logical and instruction inconsistencies.Confidence can guide requests for a second opinion but cannot establish correctness.Source: §5 · p. 8
  • Cascades outperform their strongest member and match or exceed gpt-6-sol on factual contradiction, factual fabrication, and context inconsistency, but offer limited gains on logical and instruction inconsistency.The best configuration varies by type, so deferral rates should be validated against the target type and the cost of additional calls.Source: §5 · p. 8
  • At 0.5–10% of gpt-6-sol's cost, decision models could support type-aware cascades for factual and context errors, with LLM judges reserved for logical and instruction inconsistencies.Future work should disentangle hallucination type from data source, target reasoning- and instruction-level errors, and learn escalation policies based on accuracy and cost.Source: §5 · pp. 8–9

Limitations

The benchmark's uneven source coverage, heterogeneous annotations, and limited prompt and deployment testing constrain how broadly its findings can be generalized.

Limitations

  • Context and instruction inconsistency each rely on one source, while differences in data construction, domain, and response length confound cross-dataset comparisons.Binary mappings across datasets with different error definitions may introduce label mismatches, leaving subgroup analyses observational.Source: Limitations · p. 9
  • Confidence values are not fully comparable because decision models return label probabilities while gpt-6-sol reports its own estimates, and one prompt per task leaves prompt sensitivity untested.Alternative prompts and, where supported, confidence-elicitation methods could clarify effects on accuracy and calibration.Source: Limitations · p. 9
  • Cascade evaluation uses fixed model versions and offline replay with within-source cross-fitting, while member selection and ordering use the full sample.Training-only selection followed by evaluation on unseen domains and live deployments would better assess accuracy, latency, and cost beyond the evaluated settings.Source: Limitations · p. 9

Statement of LLM Use

The authors disclose using LLM-based tools for coding assistance and manuscript language editing, while retaining responsibility for the paper's content.

Statement of LLM Use

  • LLM-based tools assisted with project-code development and refinement and with language editing of manuscript drafts; the authors take responsibility for the code, analyses, and conclusions.The editing assistance included grammar correction, sentence restructuring, and clarity improvements.Source: Statement of LLM Use · p. 9

A Benchmark Construction, Labels, and Prompts

The benchmark maps six datasets to binary labels for five hallucination types and evaluates decision models using shared, type-specific prompts. The appendix documents dataset construction, labeling choices, model configurations, and cost and latency measurement.

A.1 Hallucination Taxonomy and Examples

  • Table 3 defines the five hallucination types and gives benchmark examples, with hallucinated content bolded.Source: Table 3 · p. 12

A.2 Dataset Versions and Construction

  • The benchmark uses local source snapshots and counts 9,852 type-specific instances, representing 8,460 distinct dataset-record pairs rather than deduplicated answers.Source: §A.2 · p. 13
  • FAVABench and HADTest supply separate factual contradiction and fabrication views, with dataset-specific rules for positive labels, negatives, exclusions, and record reuse.FAVABench records carrying both types are positive in both views; 268 FAVABench and 1,124 HADTest records are shared across types.Source: §A.2 · p. 13
  • The faithfulness and instruction datasets use annotated spans, contradictions, reasoning-chain labels, or violated constraints to define their type-specific evaluation views.ROSCOE’s whole-chain label also includes incoherent reasoning; IF-RewardBench’s English filter is a preprocessing heuristic.Source: §A.2 · p. 13

A.3 Binary Label Mapping

  • Items are labeled H when the target error is annotated and N when it is absent; N does not mean the output is free of other errors.The benchmark retains each dataset’s original evaluation unit and does not rebalance classes.Source: §A.3 · pp. 13–14

A.4 Decision-Model Prompts

  • Decision models receive a shared request structure with a type-specific H/N question, criteria, and item state, alongside prompts tailored to each error type.The prompt criteria distinguish, for example, contextual faithfulness from real-world factual contradiction and instruction compliance.Source: §A.4 · pp. 14–15

B Model Configurations, Cost, and Latency

C Per-Dataset Results

The section reports balanced accuracy and error rates by source dataset, then breaks down error detection by source-annotation attributes.

C Per-Dataset Results

  • Table 7 reports each judge’s balanced accuracy by source dataset, and Table 8 reports the corresponding false-negative and false-positive rates.Source: C Per-Dataset Results · p. 16

D.1 Errors by Source Annotation

  • Table 9 breaks down H recall or, where both classes are present, balanced accuracy by error density, salience, task form, and response generator.The attributes come from source annotations and were not shown to judges.Source: §D.1 · p. 16

D.2 Error Patterns and Overlap

Error patterns vary with the evidence and task form, and decision-model errors overlap only partly with those of the generative judge.

D.2 Error Patterns and Overlap

  • 57% to 96% of pplx’s RAGTruth recall and 46% to 74% of its IF-RewardBench recall correspond to items with one versus at least three errors or violated constraints.For pplx, recall also falls from 71% to 42% on subtle RAGTruth spans and from 91% to 32% on replacement-based ContraDoc contradictions.Source: §D.2 · p. 16
  • All decision models reach only 47–63% balanced accuracy on multi-turn instructions, and every judge is near chance on non-arithmetic ROSCOE chains.On RAGTruth data-to-text items, pplx reaches 86% balanced accuracy against 79% for gpt-6-sol.Source: §D.2 · p. 16
  • pplx and gpt-6-sol are both wrong on 4% of context-inconsistency items, 6–8% of factual-error items, and 11–13% of logical- and instruction-inconsistency items.Their error-set Jaccard overlap ranges from 0.13 on context inconsistency to 0.50 on logical inconsistency.Source: §D.2 · p. 16

E Confidence Analysis Details

Confidence behavior varies by hallucination type, and cascade deferral rates affect accuracy differently across types.

E Confidence Analysis Details

  • Confidence’s association with decision correctness differs by hallucination type for every decision model (Wald χ2 = 57–142, all p < 10^-10).Source: E Confidence Analysis Details · p. 17
  • Only 10% of clef’s errors fall at confidence ≥0.9, while gpt-6-luna and especially gpt-6-sol have error distributions peaking in the highest confidence bin.For the GPT-6 models, high confidence alone does not reliably distinguish correct decisions from errors.Source: E Confidence Analysis Details · p. 17
  • The ECE model ranking is unchanged across the tested binning schemes and debiased ECE, while changing the probability-clipping threshold shifts median calibration slope by at most 0.09.Source: E Confidence Analysis Details · pp. 17–18

F Cascade Deferral-Rate Sensitivity

  • 82.2%, 84.6%, and 84.6% balanced accuracy are reached on FC, FF, and CI at intermediate first-stage deferral rates; larger rates eventually reduce performance.LI performs best without deferral, while II gains at most 0.7 points over its strongest member and remains 7.1 points below gpt-6-sol.Source: F Cascade Deferral-Rate Sensitivity · p. 18

F.1 Peak and Cross-Fitted Results

Cross-fitting preserves gains over the strongest single member for FC, FF, and CI, while LI and II show no clear gain. Peak cascade settings are post-hoc maxima, not held-out results.

F.1 Peak and Cross-Fitted Results

  • Rates and confidence thresholds are fit on four of five label-stratified folds, then applied to the held-out fold across seeds 0–4.Members remain ordered by full-sample BAcc, so cross-fitting checks rate and threshold selection, not the full model-selection pipeline.Source: §F.1 · p. 18
  • Cross-fitting lowers BAcc by at most 0.52 points; FC, FF, and CI retain gains over the strongest member, while LI and II show no clear gain.FC, FF, and CI cross-fitted differences from gpt-6-sol have intervals containing zero.Source: §F.1 · p. 18, Table 10 · p. 19
  • Peak settings maximize pooled BAcc across 36 rate configurations, but are evaluated on the same items and are not held-out results.Figure 8 varies r1 on the x-axis and fixes r2 by colored line; stars mark the highest BAcc, with gpt-6-sol and strongest-member-alone baselines shown.Source: Figure 8 · p. 19
  • Table 10 reports BAcc and paired differences; its intervals use 2,000 paired item-bootstrap resamples and are not selection-adjusted for peaks.Cross-fitted BAcc averages five seeds, and intervals condition on selected members and recorded predictions.Source: Table 10 · p. 19

Related papers

Loading 2610.37229v1…