RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

1. Introduction

• Frontier calls are metered and rate-limited, and send text to an endpoint outside the deployer’s control.Aggregation is pursued as an alternative that removes the frontier model from the loop without training model weights.Source: §1 · p. 2
• Prior evidence leaves open when a cheap panel pays, because correlated judge errors can undermine the usual benefits of aggregation.The paper focuses on the distribution of member competence and how far their errors decorrelate.Source: §1 · pp. 2–3
• RAIM pairs a cross-fitted stacked meta-learner, which jointly fits member weights, with an admissibility test read from the members’ own verdicts.The test identifies when aggregation is worthwhile; the study uses ten 4–9B judges from disjoint families across eight benchmarks.Source: §1 · pp. 2–3
• Aggregation pays when several judges are capable and err on different items; it falls back on the leader when one dominates or members are weak and err together.The authors report improvement over the best member on all four core grounded sets, statistically resolved on one, with no clear worsening.Source: §1 · pp. 2–3
• 93% of Claude Sonnet’s κ is retained at the median, though the panel falls materially behind on two of eight benchmarks.It remains competitive with purpose-trained detectors on their home benchmarks and leads them elsewhere.Source: §1 · p. 3
• At a sixty-fourth of the frontier judge’s inference cost, the main expense is one-time in-domain calibration: 50 records beat free majority voting on seven of eight sets.The reported break-even against the frontier judge is 40,000–81,000 evaluated items.Source: §1 · p. 3

2. The RAIM protocol

• The evaluation covers grounded faithfulness against references and ungrounded factuality against world knowledge, using class-balanced datasets grouped by source record or grounding document.The eight-benchmark suite has six grounded and two ungrounded sets; CNN and ExpertQA are excluded from the primary four-set grounded denominator for measurability and low-headroom reasons.Source: §2 · pp. 3, 5
• The stacked logistic regression takes member verdicts as features and gold labels as targets, fitting weights jointly so redundant echoes can be discounted.Five-fold cross-fitting produces out-of-fold predictions; abstentions are imputed as 0.5 and verdicts use a 0.5 threshold.Source: §2 · p. 4
• Comparators range from label-free majority, average-member, and Dawid–Skene baselines to global and cross-validated best singles, an undeployable test-set oracle, and a frontier judge.The ladder is ordered by the labelled information each comparator consumes.Source: §2 · p. 4
• A win requires the stacked κ to beat the CV-best single by a paired difference whose lower bound exceeds zero; a clean win also beats the oracle.Paired cluster-bootstrap intervals use 2,000 resamples; differences excluding zero are called clear, while intervals spanning zero are unresolved.Source: §2 · pp. 4–5
• +0.032 κ is the stacked panel’s paired difference over the best single judge on MedHallu, with a 95% interval of [+0.004, +0.060].The marginal score intervals largely overlap, while the paired difference excludes zero.Source: §2 · p. 4
• Ten 4–9B open-weight judges from disjoint model families score each item using a dataset-fixed prompt frame shared with the frontier judge.The panel uses question-answering or claim-support frames according to the dataset; unparseable verdicts count as abstentions.Source: §2 · p. 5

3.1. The panel against its own naive vote.

• Naive majority trails the best single by 0.094–0.228 κ on four datasets; stacking removes clear deficits and gains up to +0.279 on CNN.The stacker’s rescue is small or absent where the naive vote is not clearly below the best single.Source: §3.1 · p. 6
• Stacking is never significantly beaten by Dawid–Skene; it clearly improves over it on RAGTruth, TruthfulQA, and CNN.Dawid–Skene estimates member reliability without gold labels, while supervised reweighting is needed when a strong member must be recovered.Source: §3.1 · p. 6

3.2. The panel against the best single judge.

• On MedHallu, stacked − best single = +0.032 κ, clear against CV-best and oracle; the other three core grounded sets improve unresolved.Source: §3.2 · p. 6
• +0.082 to +0.261 κ over the label-free average member on every dataset; against CV-best, the panel improves on seven of eight, clearly on one core set.Source: §3.2 · p. 7

3.3. The competence/correlation account.

• The panel’s edge is largest with several capable members and decorrelated errors, shrinks with moderate redundant competence, and vanishes when one member dominates.This same ordering governs the frontier comparison; the advantage over the average member stays large wherever there is signal.Source: §3.3 · pp. 7–8
• At (κ0, c, ϕ0) = (0.30, 1/2, 0.40), the test admits MedHallu, WiCE, and FActScore; calibration calls reproduce 85.5% with 50 labels and 91.9% with 100.The test requires enough members at κ ≥ 0.30 and mean pairwise error-correlation no greater than 0.40.Source: §3.3 · p. 8

• Two of eight paired differences improve, with ExpertQA clear at Δκ = +0.070; six worsen, but only RAGTruth and TruthfulQA are materially adverse.The MedHallu worsening is 1.8 balanced-accuracy points, which the authors characterize as too small to matter in deployment.Source: §4 · pp. 8–9
• 73.0 versus 75.9 balanced accuracy: the panel gives up 2.9 points on average, retains median 93% of Sonnet’s κ, and materially trails on two benchmarks.Source: §4 · p. 9
• At $0.039 per 1,000 items versus Sonnet’s $2.52, the panel’s $100–$200 calibration outlay is repaid after about 40,000–81,000 items.The panel runs on premises, so evaluated text stays within the deployer’s infrastructure and avoids rate limits and model deprecation.Source: §4 · p. 9
• On four LLM-AggreFact sets, the panel reaches 69.5 bacc, within the agreement band of the top system (71.4) and GPT-4o (70.8).MedHallu levels do not license rankings because abstentions are charged and the compared systems are not evaluated on the same items; RAGTruth prevalence also differs.Source: §4 · pp. 9–10
• Larger open judges do not improve monotonically with scale; Qwen2.5-32B is close on average at 72.6 versus 73.0 suite bacc, but its task-level fit is not known in advance.The panel’s regime can instead be read from its members.Source: §4 · p. 10

5. Conclusion

• Across eight benchmarks, the panel gives a verdict at least as good as its best member, retains median 93% of Sonnet’s κ, and gives up 2.9 balanced-accuracy points on average.The panel clearly leads on one benchmark and falls materially behind on only two; a 50–100-record calibration is amortized after about 40,000–81,000 items.Source: §5 · p. 10
• The panel’s gain requires several capable members with decorrelated errors; the study characterizes one fixed roster of frozen generalists, and roster or model-version changes may shift that condition.The authors identify roster composition, rather than simply adding larger members, as the lever to pull.Source: §5 · pp. 10–11

Code and data availability

• Two CC BY-NC-SA repositories release the inference and analysis pipelines and derived verdicts; analysis regenerates tables without a GPU or network, while inference is not bit-reproducible across stacks.Source: Code and data availability · p. 11

Ethics statement

• MedHallu contains synthesized medical hallucinations, and the evaluated detector is a research artifact, not a clinical tool.Source: Ethics statement · p. 12

Appendix A. Roadmap to the appendices

• Appendices B–E extend the evidence from item-level testing to transfer; F–H compare outside systems and calibrations, while I–M provide tables, protocols, and reference material.The roadmap maps appendices to the main-text sections they support.Source: Appendix A · p. 17, Table 3 · p. 17

The competence/correlation account in full

• The panel never recovers any of 529 leader-error items when every member shares the error, but recovers about six in seven when nine members are correct.Across 1,998 items, recovery correlates +0.41 with the fraction of members correct (p < 10^-20); the relation is positive in all eight datasets, with median +0.48.Source: Table 4 · p. 18, Appendix B · p. 18
• The cross-dataset predictor has the predicted but nonsignificant association with gain (Spearman ρ = +0.47, p = 0.24), and separate-coordinate patterns are sensitive to held-out sets.CNN combines the suite’s highest error-correlation with a +0.051 gap whose interval accommodates either sign; no Table 5 entry reaches significance at n ≤ 8.Source: Appendix B · pp. 18–19
• The panel’s gain is largest when several capable members have decorrelated errors, unresolved when competence is moderate and redundant, and negligible when one judge dominates a weak field.FActScore, MedHallu, and WiCE are the most panel-favourable; RAGTruth and TruthfulQA are leader-dominated, while CNN and ExpertQA have scant competence and highly correlated errors.Source: Appendix B · p. 19
• Dropping members confirms that gains are jointly held in favourable regimes, while Gemma dominates TruthfulQA and RAGTruth and stands out on WiCE and CNN.On MedHallu no member is worth more than 0.006 κ; on FActScore five members each cost 0.03 or more, while dropping Gemma costs 0.066 κ on TruthfulQA and 0.063 κ on RAGTruth.Source: Appendix B · p. 20
• Adding members helps on low-correlation sets, leaves leader-dominated and redundant sets flat, and peaks early before diluting on WiCE.The panel-size sweep is presented as a diagnostic, not a recipe for choosing the number of members.Source: Appendix B · p. 20
• The ten-member panel is worth between 1.15 and 5.11 independent votes across datasets, exceeding the reported frontier-panel comparison on four of eight datasets.The cross-study comparison is of a derived panel-internal quantity, not scores, and the studies use different tasks and no shared rows.Source: Appendix B · pp. 20–21

The admissibility test

• The rule requires enough competent members and sufficiently low mean pairwise error-correlation; its reported instance uses κ0 = 0.30, c = 0.40, and ϕ0 = 0.40.At this instance, the eight full-panel calls coincide with the competence threshold alone; the correlation condition is not tested there.Source: Appendix C · pp. 21, 24
• The test admits MedHallu, WiCE, and FActScore; the other five datasets are rejected, and the suite’s clear improvement over the CV-best single occurs within the admitted set.The test is a decision heuristic, not a hypothesis test, and has no interval of its own.Source: Appendix C · p. 22
• Across all 396 threshold pairs that split drawn sub-panels, admitted panels more often stay within 0.10 κ of the frontier and beat their best member than rejected panels.The direction holds for all 375 threshold pairs splitting the eight full panels; these configurations share members and items, so they are not evidence across tasks.Source: Appendix C · p. 23

The labelling budget

• The test’s full-sample call is reproduced an average 85.5% of the time from 50 labelled records and 91.9% from 100.FActScore and TruthfulQA lie near the decision boundary and are correspondingly unstable.Source: Appendix C · p. 24
• The budget at which the stacker overtakes the vote ranges from 10 records on RAGTruth and CNN to the full 132-record pool on FActScore.ExpertQA continues to gain through its full pool of 1,088 records; the vote is already within 0.018 κ of full supervision on FActScore.Source: Appendix C · p. 25
• Fifty labelled records let the stacker beat the label-free majority vote on seven of eight datasets; at ten records, it falls below the vote on six.FActScore is the exception at 50 records; ExpertQA needs more labels, while CNN’s training pool is too small for the 100-record point.Source: Appendix C · p. 26

Base-versus-instruct probe

• Instruction tuning lowers mean within-panel error-correlation on seven of eight datasets, with shifts from −0.085 on WiCE to −0.495 on FActScore; MedHallu’s +0.018 shift includes zero.The comparison uses eight public base checkpoints and their instruction-tuned variants under matched constrained decoding; after excluding Falcon, the shift remains clearly negative on six datasets.Source: Appendix D · p. 26

Measurement and mechanism

• Leave-one-dataset-out transfer yields zero wins in four core grounded sets, with significant in-domain-minus-transferred κ gaps of +0.111 on MedHallu and +0.175 on RAGTruth.On MedHallu, transfer falls below the best single by −0.079; on RAGTruth, an unresolved in-domain improvement becomes a transfer loss of −0.165.Source: Appendix E · p. 27
• The stacker learns weights from the training pool, but the most reliable judge changes by dataset, so fixed off-domain weights can underweight the held-out leader.On MedHallu, the in-domain leader is Granite, yet the transferred fit gives Granite little weight against Gemma.Source: Appendix E · p. 27

Scope of the negative result

• The negative transfer result is limited to generative panels using hard verdicts and a stacker pooled over the other three datasets.The study does not establish that every panel fails to transfer; transfer with members placed on a common scale across domains remains untested.Source: Appendix E · pp. 27–28

Shared rows and metric conversion

• On the four LLM-AggreFact sets, the comparison uses the same underlying rows and balanced accuracy; for full-coverage judges on class-balanced sets, κ = 2 bacc −1.Abstention affects κ through the committed-set treatment, while balanced accuracy remains comparable in expectation after class balancing.Source: Appendix F · p. 28, Table 8 · p. 29

Harness calibration

• Two harness checks agree with published balanced accuracy to about one point: MiniCheck-Flan-T5-L scores 68.4 versus 68.8, and Qwen2.5-72B scores 70.4 versus 69.2.The authors treat differences under three points between blocks as uninformative.Source: Appendix F · p. 28

Summary of the comparison

• Across the published comparisons, systems clearly ahead of the panel were trained on the target corpus or received evidence withheld from the panel; the prompted frontier anchor on MedHallu matches their frontier judge.ExpertQA’s external leaderboard ceiling also aligns with the panel’s low-headroom result: 61.8 bacc for the panel versus a best leaderboard score of 60.9.Source: Appendix F · p. 30

Comparison caveat

• Cross-paper levels remain difficult to compare because published numbers use different prompting and inference protocols, which can make factuality metrics fragile.Within-suite contrasts use one frame per dataset, but those controls do not carry across papers.Source: Appendix F · p. 30

Frame comparison

• Changing the four claim-verification sets from the question–answer frame to claim-support framing raises the unweighted panel by 10.7 balanced-accuracy points on average.Under the revised frame, the frontier judge’s κ rises on all four sets; the authors attribute the shift to task scaffolding rather than a change in the judge’s ability to read evidence.Source: Appendix F · p. 31

Appendix G. Purpose-trained cheap judges off their home benchmark

• The two MiniCheck specialists verify claims against documents, so they apply only to the six grounded sets, while the evaluation-tuned judges use native graded interfaces thresholded at each scale’s midpoint.The reference slot describes an ideal faithful answer rather than supplying a gold label.Source: Appendix G · pp. 32–33
• The evaluation-tuned judges are off-the-shelf baselines, not faithfulness specialists; their documented midpoint threshold is untuned, and their scores may be depressed by shorter context windows.Only Prometheus-2 reaches the panel’s and frontier judge’s 8192-token window; Auto-J and JudgeLM are capped at 4096 and 2048 tokens, respectively.Source: Appendix G · p. 32
• Granite Guardian can be run on FActScore and TruthfulQA, but its faithfulness-related risks require retrieved context, so it cannot provide the measured faithfulness judgment on ungrounded items.Its other listed risks can be scored from the prompt and response alone.Source: Appendix G · p. 32
• Purpose-trained judges are compared with the panel, its best single member, and the frontier judge on identical balanced instances under each dataset’s scoring frame.The five judges’ Cohen’s κ values are reported in Table 12; coverage ranges from 0.892 to 1.000 across judge/dataset cells.Source: Table 12 · p. 33

Scaling

• The scaling comparison uses 4-bit awq Qwen2.5-32B and Qwen2.5-72B under the panel’s protocol, but its conclusion is scoped to that quantisation.The authors report that Qwen2.5-72B reaches 70.4 mean balanced accuracy on four LLM-AggreFact sets, against 69.2 published for the unquantised checkpoint.Source: Appendix H · p. 33
• Qwen2.5-72B trails Sonnet on four of eight sets, matches or exceeds it on four, and remains well behind on RAGTruth, FActScore, MedHallu, and CNN.On WiCE, XSum, and ExpertQA its κ values exceed Sonnet’s (0.629 vs. 0.548, 0.446 vs. 0.418, and 0.202 vs. 0.165); on TruthfulQA they are 0.630 vs. 0.623.Source: Appendix H · pp. 33–34
• Qwen2.5-32B and Qwen2.5-72B are the panel’s closest rivals, each scoring 72.6 suite balanced accuracy versus the panel’s 73.0; the open-model ladder is not monotone in size.Qwen2.5-32B costs $0.008 per 1,000 items, compared with the panel’s $0.039.Source: Appendix H · pp. 33–34

Cost

• The panel costs $0.039 per 1,000 items against Sonnet’s $2.52, about a sixty-fourth, using sequential member inference and a public A40 rental rate.Sequential timing is pessimistic for latency when members can run in parallel; no rate is quoted for Qwen2.5-72B because it does not fit the same hardware envelope.Source: Appendix H · p. 34

Amortised cost and break-even

• The panel’s amortized cost per 1,000 items is 0.039 + 103C/N and undercuts the frontier above N ≈103C/2.48.At a $100 calibration budget, the stated break-even is about 40,000 items; the label price assumes two minutes of annotation at $30 per hour.Source: Appendix H · p. 34
• A cheap panel pays for sustained evaluation workloads rather than occasional ones because its one-time calibration outlay is amortized across items.The break-even estimate varies with annotation time and workload size; reducing or replacing gold labels is not priced.Source: Appendix H · p. 34

Full method ladder

• The appendix provides full per-dataset ladders for Cohen’s κ and balanced accuracy, from label-free aggregators and single judges through the stacked panel.Tables 15 and 16 report central estimates; 95% cluster-bootstrap intervals are in the released artefacts.Source: Appendix I · p. 35, Table 15 · p. 36, Table 16 · p. 36

Aggregator contrasts

• Table 17 isolates the stacked panel’s contrast with naive voting and unsupervised Dawid–Skene, using the CV-best single judge as the shared baseline.This makes the two aggregation comparisons directly comparable against the same single-judge reference.Source: Appendix I · p. 35

Member-level results

• Gemma is strongest on average and receives the largest stacker weight on five of eight sets, but Granite leads on MedHallu and Qwen on XSum.The section presents member agreement and stacker coefficients together; individual coefficient ranks are noisy across folds, so broad column structure is more informative than fine rank differences.Source: Appendix I · p. 36, Figure 10 · p. 37
• Figure 10 pairs each member’s Cohen’s κ against gold with its mean cross-fit stacker coefficient across datasets.Rows follow roster order and datasets descend by mean single-member competence; coefficient ranks are noisy across folds.Source: Figure 10 · p. 37

The unit of judgement

• Each judged item is one prompt and one verdict, and the exact 50/50 class balance makes the κ = 2 bacc −1 identity exact.The cluster bootstrap resamples the source-record or grounding-document clusters, not pairs shown jointly to judges.Source: Appendix J · p. 38

Pairing on the four question-answering sets

• Question-answering datasets pair one supported and one hallucinated item within a source record, yielding two-item clusters.MedHallu supplies the pair directly; RAGTruth, TruthfulQA, and FActScore use dataset-specific selection procedures.Source: Appendix J · p. 38

Pool balancing on the four claim-verification sets

• On the four claim-verification sets, claims are balanced across supported and unsupported pools because almost no document supplies both classes.The minority pool is retained in full and the majority pool is sampled to match it; selected claims are then grouped by shared document.Source: Appendix J · p. 38, Table 18 · p. 39
• Claim-verification clusters contain one to four claims of either class, and CNN’s n = 114 reflects severe upstream class imbalance.Because most documents contribute one claim, cluster resampling is close to item resampling on these four sets, while it binds fully on the other four.Source: Appendix J · p. 39

Choice of one pair per record

• Taking one pair per record gives records equal weight, but discards annotations that could sharpen the within-record picture.Scoring all available candidates would multiply items without multiplying clusters, allowing verbose records to carry more weight in point estimates.Source: Appendix J · p. 39

Prompt and verdict frame

• The four question-answering prompts define supported answers as fully consistent with the truth or evidence and ask for a binary hallucination verdict.For LLM-AggreFact, the prompt instead frames the task as whether each claim is supported by or directly entailed by its document.Source: Appendix K · pp. 40, 42

Roster and prompts

• The roster fixes ten instruction-tuned open-weight checkpoints from distinct developers, and each dataset uses the same fixed prompt for every judge.The roster is held fixed across datasets and experiments; members are identified by their full checkpoint names.Source: Appendix K · p. 40, Table 21 · p. 41

Coverage

• Across the suite, abstentions account for about 2.1% of ten-judge votes; the stacker imputes them as 0.5, while majority vote uses only committing members.Every member exceeds 0.95 coverage on XSum, CNN, FActScore, and TruthfulQA, while some members fall below that threshold elsewhere.Source: Appendix K · p. 42

Abstention encoding

• All eight abstention-encoding comparisons are unresolved; the largest absolute difference is 0.009 κ, and the mean is −0.001.On MedHallu, the interval excludes effects beyond 0.010 κ; WiCE’s interval is uninformative because its 8.1% abstention rate occurs on 222 items.Source: Appendix K · p. 43

Multiplicity

• The Holm correction preserves MedHallu’s win among the four core grounded sets, but not when all eight datasets form one test family.The adjusted p-value is 0.048 for the core four and 0.096 across all eight; the correction changes no reported verdict.Source: Appendix K · pp. 43–44

Missing-vote denominator confound

• On MedHallu, restricting the panel–Granite comparison to Granite’s 1,653 answered items leaves the κ gap essentially unchanged: +0.031 [+0.003, +0.058].The full-denominator gap is +0.032 [+0.004, +0.060]; because Granite abstains disproportionately on harder, hallucinated items, the reported gap is conservative.Source: Appendix K · p. 44

Panels and correlated errors

• Prior panel studies often ask whether substitution pays on average; RAIM instead characterizes the member-competence and error-correlation regimes in which it pays.The review notes that a panel’s errors can be correlated, so disjoint model families do not guarantee useful diversity in advance.Source: Appendix L · pp. 44–45

Noisy-voter aggregation

• Unsupervised noisy-voter methods infer reliability from consensus, which can be shared error on a correlated panel; dependence-aware extensions make additional assumptions.The review leaves open whether those extensions recover competence on panels as correlated as these judges.Source: Appendix L · p. 45

Correlated errors

• Prior work reports that judges’ errors overlap and that correlation rises with model accuracy, motivating measurement of panel competence and error correlation rather than assuming diversity.Disjoint families provide no guarantee of diversity a priori.Source: Appendix L · p. 45

Where this work sits

• RAIM addresses whether a local cheap panel can replace a paid frontier judge, including calibration cost and advance admissibility, rather than only how to aggregate a panel already chosen.Its account reports a median 93% of the frontier judge’s κ at one sixty-fourth of its inference cost, with calibration break-even at about 40,000–81,000 items.Source: Appendix L · p. 46

Cascading and routing

• Cascades retain a frontier judge for uncertain items, so the review treats them as complementary when constraints require on-premises inference or avoid rate limits.A panel-first cascade is described as a natural extension, while a cascade may be preferable when cost is the only constraint.Source: Appendix L · p. 47

Purpose-trained judges

• Purpose-trained specialists use labels to fine-tune one evaluator, whereas RAIM uses labels for a second-stage aggregator over frozen generalists.Specialists can hold an advantage on their home reference-conditioned benchmarks but cannot score ungrounded tasks; cross-paper protocol differences also limit direct comparisons.Source: Appendix L · p. 47

Costing

• Cost and break-even estimates use reference GPU rates, one-date API list prices, and a $1 label cost, so the crossing volume varies with those assumptions.Expert annotation can raise the label cost, and the estimate excludes orchestration costs, which may appear as latency for sequential serving.Source: Appendix M · p. 48

Frame sensitivity and quantisation

• The fixed per-dataset frame is not tuned per judge, and a frontier-specific prompt search could change the ExpertQA reversal; open-judge scaling results are scoped to 4-bit AWQ.The matched-label comparison of few-shot frontier exemplars was not run.Source: Appendix M · p. 48

External comparisons

• External comparisons use published results under differing protocols and, for some datasets, different slices or prevalence, making them weaker evidence than within-suite contrasts.One rerun checkpoint was not reproduced, and the effective-independence comparison also spans studies with different tasks, gold labels, and panel sizes.Source: Appendix M · p. 48

Scope of the panel and competitors

• The comparison covers binary verdicts from one fixed roster of frozen generalists against Claude Sonnet, not alternative rosters, matched-budget specialists, or frontier judges broadly.The work also does not measure a panel-first escalation cascade or the share of frontier calls it would retain at matched quality.Source: Appendix M · p. 48

Class balance

• All datasets are exactly balanced, so the reported κ readings and admissibility rule apply to balanced evaluation, not directly to streams with different class prevalence.The paper does not evaluate either shifting the stacker intercept or drawing calibration data from deployment prevalence.Source: Appendix M · p. 49

Evidential base

• The account’s cross-dataset ordering rests on eight datasets, and the same eight formed the account and admissibility rule, leaving the rule consistent with rather than validated by its evidence.Whether the two statistics retain their interpretation beyond English or beyond the represented domains is untested.Source: Appendix M · p. 49