PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Probes: Generator-Independent Hallucination Detection from Observer Model Hidden States

Michael Rathmayr, Ádám Kovács, Gábor Recski

arXiv:2610.04642v1cs.CL

TL;DR

RAG responses can contain unsupported claims, while existing hidden-state probes depend on the generating model and cannot reach closed-weight generators. This paper probes a separate observer that reads the context, question, and response without generating; its probes perform strongly across observers, and blending with a span detector improves on that detector alone.

  • Problem

    Existing hidden-state hallucination probes use the generating model’s activations, so changing generators invalidates the probe and closed-API generators are inaccessible to it.

    Source: §1 · p. 1

  • Method

    A frozen observer reads the context, question, and response in one forward pass, and logistic regression over its hidden states estimates whether the response is unsupported.

    Source: §3 · p. 3, §3.1 · p. 3

  • Results

    0.879–0.894 AUROC across observers and 0.924 blended; the paper reports that blending improves ranking and threshold behavior over the span detector alone.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    One fitted probe is evaluated across six generators, including generators whose internals are unavailable to the detector.

    Source: §1 · p. 2, §3.1 · p. 3

    The hold-out controls bound the cost of removing a generator at roughly 0.02 AUROC rather than showing it is zero, and per-generator confidence intervals are wide.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Detecting responses that retrieval-augmented generation does not ground in its context trades speed against accuracy: surface checks miss paraphrased fabrication, sampling-based methods cost extra generations. Hidden-state probes sit between the two, but every existing one reads the generating model's own activations, so a change of generator invalidates the detector and a closed-weight generator is out of reach. This paper removes that coupling. The Grounding Probe is logistic regression over the mean-pooled middle-layer hidden states of an observer language model that reads the context, question, and response in one forward pass and generates nothing, with the recipe it needs: pool over response tokens, read a middle layer, and control capacity, which closes the train-test AUROC gap from 0.087-0.202 to 0.009-0.013. Asking the observer outright, rather than reading its hidden state, costs at least +0.166 AUROC in every one of four models. Fitted on 15,090 annotated responses it reaches 0.879-0.894 AUROC on RAGTruth test across four observers, and 0.924 AUROC with 0.820 F1@0.5 averaged with a supervised span detector, 0.060 above that detector alone. One probe holds across six generators, and hold-out controls, including one in which no evaluation prompt appears in training, bound the cost of removing a generator at about 0.02 AUROC. Code, probes, and predictions are released.

1 Introduction

RAG responses can remain unsupported despite retrieval, while current checks trade speed against sensitivity or depend on the generating model’s hidden states. The paper proposes probing a separate observer’s representations so one detector can serve different generators, including closed-API systems.

1 Introduction

  • RAGTruth includes unsupported content in 9.3% of GPT-4 responses and in more than 50% for weaker generators, motivating checks that fit within serving latency.The paper identifies deciding whether each response is grounded as a production-system requirement.Source: §1 · p. 1
  • Existing hidden-state probes depend on the generator’s activations, while surface classifiers can miss fluent paraphrases and sampling-based checks cost five to ten times one generation.The paper frames observer probing as an intermediate option: one forward pass reads what the observer’s activations encode.Source: §1 · p. 1
  • A separate observer reads context, question, and response without generating, allowing one hidden-state probe to serve any generator, including closed-API systems.The Grounding Probe reads mean-pooled middle-layer states; the generator contributes only its already-produced text.Source: §1 · p. 1, Figure 1 · p. 2
  • The proposed recipe uses response-token mean pooling, a middle extraction layer, and capacity control; 15,000 annotated responses suffice, while an uncontrolled probe memorizes them.The projection controls capacity without tuning, although tuned regularization is marginally ahead within the prespecified 0.015 AUROC margin.Source: §1 · p. 2
  • Averaging the probe score with a supervised span detector improves ranking and threshold behavior over that detector alone, without retraining it.The paper evaluates one fitted probe across six RAGTruth test-split generators.Source: §1 · p. 2

2 Related Work

Prior work approaches hallucination detection through supervised span prediction, text-based entailment, and probes of model internals. The paper identifies probing a separate evaluator model as an unreported combination and also studies combining response- and token-level detectors.

2 Related Work

  • Supervised span detectors predict unsupported spans, while entailment and alignment models assess text directly and remain generator-independent but limited to surface evidence.RAGTruth provides word-level annotations over close to 18,000 responses.Source: §2 · p. 2
  • Internal-probing work includes truthfulness, uncertainty, and RAG methods; ReDeEP and LUMINA score reliance on retrieved context against parametric knowledge.Reported approaches range from single-layer extraction to multi-layer classifiers and detection during generation.Source: §2 · p. 2
  • Prior internal probes extract activations from the generating model, leaving them tied to that generator and unavailable for closed-weight generators; the paper reports probing a separate evaluator instead.The authors state they are unaware of prior work combining separate-evaluator assessment with internal probing.Source: §2 · p. 2
  • Detector-combination studies are usually isolated across different splits and generators, leaving error correlation between method families unmeasured; this paper tests blending and routing.The authors report that routing underperforms the simpler averaging alternative.Source: §2 · p. 3

3 Grounding Probes

The Grounding Probe uses a frozen observer to read a retrieval-augmented response and applies logistic regression to its hidden states. Its setup, representation choices, fitted transformations, training data, and score blending define the deployed procedure.

3 Grounding Probes

  • A Grounding Probe is logistic regression over hidden states from an observer that reads a context-question-response triple without having generated it.The generator appears only through the text it already produced, so generator independence follows from the setup.Source: §3 · p. 3

3.1 Observer setup

  • The frozen observer runs one forward pass with generation disabled, and that pass is the entire inference cost.The probe can score responses from any generator, including one behind a closed API.Source: §3.1 · p. 3

3.2 Hidden state extraction

  • The probe mean-pools response-token hidden states and excludes context and question positions because those tokens are shared by faithful and unfaithful responses using the same passage.The selected extraction layer is determined separately in Section 6.1.Source: §3.2 · p. 3

3.3 Classifier

  • The classifier normalizes dimensions using training-set statistics, projects to 512 leading principal components, and maps the result to hallucination probability with logistic regression.The composed deployed probe is an affine functional of the raw hidden state, with inference reduced to one dot product.Source: §3.3 · p. 3

3.4 Training

  • Probes are fitted on 15,090 RAGTruth training responses, collapsed from human span annotations to a binary response label indicating whether any span is unsupported.Fitting requires one observer forward pass per sample followed by convex optimization.Source: §3.4 · p. 4

3.5 Score blending

  • The blend combines the probe score with a token-level span score max-pooled to response level, using α = 0.45 throughout.The authors choose blending over confidence routing because routing costs F1 for structural reasons rather than a poor threshold choice.Source: §3.5 · p. 4

4 Experimental Setup

The experiments fit probes on RAGTruth training data and evaluate them on a held-out test split balanced across task types and generators. The setup also reports evaluation-set selection decisions and defines the populations used for performance and latency measurements.

4 Experimental Setup

  • RAGTruth supplies 15,090 training responses and a 2,700-response test split with 34.9% hallucinated responses, balanced across three task types and six generators.HaluEval contributes 30,000 additional samples used only for transfer measurement.Source: §4 · p. 4
  • Four observers—Qwen2.5 3B, 7B, and 14B and Llama 3.1 8B—are compared with two LettuceDetect sizes, using matched fp16 fitting and inference precision.Accidental precision mismatches during development produced AUROC drops of 0.02–0.16.Source: §4 · p. 4
  • AUROC is the primary threshold-free metric; F1 uses the fixed 0.5 threshold, and intervals use 10,000-resample percentile bootstraps, paired for differences.The fixed F1 threshold avoids tuning on the test split.Source: §4 · p. 4
  • The headline blending weight α was selected on the evaluation split: α = 0.50 yields 0.9236 AUROC versus 0.9240 at α = 0.45, a 0.0004 difference.F1@0.5 is 0.804 at α = 0.50 versus 0.820 at the selected α; an honest treatment requires a third split and fresh extraction.Source: §4 · p. 4
  • Choosing among 84 α–observer combinations on one split overstates held-out performance by 0.002 AUROC and 0.007 F1@0.5; reported intervals are conditional on selection.The headline also selects the best of eight detector–observer pairings from a split with about 944 scored configurations.Source: §4 · p. 4
  • PCA-512 was selected using held-out test AUROC, but its margin over PCA-256 is at most 0.007 AUROC and paired intervals overlap zero for three of four observers.The authors characterize capacity control as necessary, projection width as not, and select the extraction layer using training-data cross-validation.Source: §4 · p. 4
  • Table 1 reports response-level detection on the 2,700-row RAGTruth test split; its lower block pools six generators, while prior-work blocks are not comparable.The caption identifies per-generator results as the matched comparison.Source: Table 1 · p. 5
  • Latency averages depend on population: the smallest observer is 49 ms versus 37 ms for the large detector pooled, but 60 ms versus 81 ms on evaluation rows.The evaluation population is 2,700 rows within a 12,700-sample benchmark run.Source: §4 · p. 5

5 Results

Blending the probe with LettuceDetect-large raises AUROC from 0.864 to 0.924 and F1@0.5 from 0.792 to 0.820. The probe performs strongly alone, complements the detector on summarization, and retains performance across generators, with held-out controls bounding the effect of removing a generator.

5.1 Grounding probes alone

  • 0.879–0.894 AUROC on RAGTruth test across four observers, with summarization lowest at 0.770–0.838 for every observer.Data-to-text reaches 0.861–0.880 and question answering 0.858–0.870; response length does not explain summarization’s lower scores.Source: §5.1 · p. 5
  • 0.418–0.728 AUROC from prompting the same observers trails their hidden-state probes’ 0.879–0.894, despite both readouts using one forward pass.The paired advantage is +0.166 to +0.470, with every interval excluding zero.Source: §5.1 · pp. 5–6

5.2 Combining with a token-level detector

  • 0.924 AUROC and 0.820 F1@0.5 for the blend improve on LettuceDetect-large’s 0.864 AUROC and 0.792 F1@0.5.The AUROC gain is 0.060, with paired interval [0.050, 0.070]; at untuned α = 0.50, F1@0.5 is 0.804.Source: §5.2 · p. 6
  • +0.092 AUROC on summarization is the probe’s largest task advantage over LettuceDetect-large, whose score is 0.746 versus the probe’s 0.838.The detector scores 0.897 on data-to-text and 0.855 on question answering, compared with the probe’s 0.880 and 0.858.Source: §5.2 · p. 6
  • 0.037 expected calibration error for the blend improves on the probe’s 0.073 and detector’s 0.101; Qwen2.5 3B achieves 0.915 AUROC and 0.809 F1@0.5 at 60 ms.The 60 ms latency is the lowest observer latency measured for the combination.Source: §5.2 · p. 6

5.3 Independence from the generator

  • 0.924 pooled blended AUROC exceeds the six-generator macro-average of 0.896 by 0.028, so base-rate differences make the pooled comparison easier.Hallucination rates range from 9.3% to 55.8%; comparisons with per-generator literature require a matched generator subset.Source: §5.3 · p. 6, Table 2 · p. 7
  • 0.858–0.934 blended AUROC across six generators comes from one probe fitted once, without per-generator tuning.Table 2 reports AUROC by generator; the pooled 0.924 exceeds the macro-average 0.896 by 0.028 as hallucination rates range from 9.3% to 55.8%.Source: §5.3 · p. 6, Table 2 · p. 7
  • -0.003 macro AUROC is the average change across five seeds when held-out generator prompts are also removed from training.Across row and context controls, intervals of about 0.02 AUROC bound the effect; the size-only-adjusted generator-presence estimate is +0.001 [-0.011, +0.013].Source: §5.3 · p. 7

6 Analysis

The probe’s performance depends on pooling response tokens, reading a middle layer, and controlling capacity; transfer and blending results also reveal important task and deployment trade-offs.

6.1 What the probe needs

  • Mean pooling over response tokens beats single-position reads at all 28 tested layer-observer combinations, by at least 0.070 AUROC.The pre-generation token scores 0.522–0.524, while the second-last token scores 0.632–0.703; pooling adds at least a tenth over the latter.Source: §6.1 · p. 7, Table 3 · p. 8
  • The best tested layer lies at 42–50% of the stack counted back from the output; the final layer costs 0.054–0.078 AUROC.The sweep tested the upper half of each stack in seven steps, so a better layer further down is not ruled out; for Llama 3.1 8B, the selected layer is the deepest tested.Source: §6.1 · pp. 7–8
  • 0.886–0.900 AUROC with tuned regularization exceeds 0.879–0.894 for PCA-512, while both control overfitting in the 15,090-sample training set.The deployed PCA-512 probe reduces the train–test gap from 0.087–0.202 to 0.009–0.013; the tuned full-dimensional probe recovers similar capacity control.Source: §6.1 · p. 8, Figure 2 · p. 9

6.3 Blending weight, and why not routing

  • Blending the tuned probe changes AUROC by only +0.001 to +0.004 across observers, with a maximum gain of +0.004.The reported latency comparison measures stages per sample on an A100; measured concurrent execution costs 1.33× the derived maximum.Source: §6.1 · p. 8, Figure 4 · p. 9, §6.4 · p. 10

6.1 What the probe needs

  • 15,090 RAGTruth samples with human labels and context outperform QA supervision by 0.080–0.208 AUROC even against the full-dimensional grounding row.The broader comparison is 0.162–0.236 AUROC, but it also differs in layer, dimensionality, precision, and context.Source: §6.1 · p. 8, Table 3 · p. 8

6.2 Cross-dataset transfer

  • On HaluEval, RAGTruth-trained probes reach 0.837–0.943 AUROC for question answering, 0.682–0.804 for dialogue, and 0.643–0.704 for summarization.RAGTruth has no dialogue data; summarization still loses 0.10–0.17 AUROC against its RAGTruth counterpart despite being represented in RAGTruth.Source: §6.2 · p. 8, Figure 3 · p. 9

6.3 Blending weight, and why not routing

  • Across α values from 0.30 to 0.50, blended AUROC varies by at most 0.004, while F1@0.5 peaks at 0.820 with α = 0.45.Routing uncertain samples to the probe raises AUROC by 0.019–0.029 over the large detector but costs 0.026–0.042 in F1 at the best threshold.Source: §6.3 · p. 9

6.4 Latency

  • Observer forward passes take 60 to 198 ms per sample on an A100, and observer size moves blend latency more than detector size.Concurrent execution costs 1.33× the derived per-sample maximum on a two-A100 host; the reported blend latency excludes coordination overhead.Source: §6.4 · pp. 9–10, Figure 4 · p. 9

7 Conclusion

The conclusion is that a model that only reads a retrieval-augmented response can support hidden-state hallucination detection without using the generator’s own activations.

7 Conclusion

  • Hidden-state hallucination detection does not require the generating model’s activations.The observer reads the response and encodes whether it is grounded.Source: §7 · p. 10
  • A probe on observer hidden states reaches 0.879–0.894 AUROC.The conclusion also reports 0.924 AUROC for the blend.Source: §7 · p. 10
  • The observer’s hidden-state probe outperforms asking that model for a verdict outright.Source: §7 · p. 10

Limitations

The reported evidence is bounded by benchmark, task, threshold, and deployment conditions, and the generator-independence controls do not establish a zero cost.

Limitations

  • Each probe is tied to one observer, extraction layer, and numeric precision; transfer across observers is untested.Swapping observers requires re-extracting hidden states and a one-time layer sweep for an unfamiliar architecture family.Source: Limitations · p. 10
  • The benchmark evidence relies largely on RAGTruth, whose shared annotation convention cannot by itself show that the probe tracks groundedness rather than that convention.HaluEval question-answering transfer argues against the narrowest version of that concern, while weaker dialogue and summarization transfer leaves it open.Source: Limitations · p. 10
  • Generator hold-out controls bound the cost of removing a generator at roughly 0.02 AUROC rather than zero.The controls concern the probe alone, matched training uses about 46% of a generator’s deployed row count, and per-generator intervals span up to 0.13 AUROC.Source: Limitations · p. 10
  • The reported blend latency is a floor: concurrent execution costs 1.33× the derived maximum, and the ratio comes from a different machine.Source: Limitations · p. 10
  • The verdict comparison uses one untuned prompt, so its +0.166 minimum AUROC gap is measured against a lower bound.The paper also does not report a generator-own-hidden-state probe under this protocol or a non-linear probe.Source: Limitations · p. 10
  • The probe’s pooled AUROC may benefit from base-rate separation across generators and task types: its task macro-average is 0.859 versus 0.894 pooled.Length accounts for little of the probe’s ranking, but other groundedness correlates have not been isolated.Source: Limitations · p. 11
  • The default threshold is unreliable across prevalence conditions: probe F1@0.5 falls to 0.404–0.405 on two low-prevalence generators, and transfer reports AUROC only.A deployment whose base rate differs from RAGTruth’s must set its own threshold.Source: Limitations · p. 11
  • Summarization is the weakest task type at 0.770–0.838, while the long-context regime, other languages, and specialized domains remain untested.The longest RAGTruth test input is roughly two thirds of the 4096-token budget, and none is truncated.Source: Limitations · p. 11

A Blending and Routing Detail

Blending improves AUROC over the token-level detector across tested pairings, whereas routing trades higher AUROC for lower F1 and has structurally bimodal scores.

Related papers

Loading 2610.04642v1…