Source-linked AI summary
When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models
Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
TL;DR
Language models may compute correct answers internally while their answer channels fail to express them, raising the question of when probing reveals genuine concealed knowledge. This paper traces verdict information from hidden states through output logits to behavior, tests the diagnosis across tasks and model configurations, and finds that threshold miscalibration explains much of the gap while probe–margin comparisons separate miscalibration, concealment, and absence.
Problem
The central problem is determining when apparent hidden knowledge reflects genuine internal computation rather than a failure or artifact of the answer channel.
Method
The paper traces exact verdicts through hidden states, output margins, emitted answers, cross-configuration evaluations, calibration interventions, and a surface-cue-free maze task.
Results
Across tasks and scales, the dominant gap is threshold miscalibration: behavior tracks threshold offset, while internal and margin rankings remain informative; calibration repairs performance.
Takeaways & Limitations
The probe–margin comparison distinguishes miscalibrated, concealed, and undetected verdicts, including correctly reporting absence on difficult maze paths.
Takeaways & Limitations
The evaluation is limited to synthetic English domains and models up to 8B, with correlational probes and some logic signal remaining candidate-intrinsic.
Abstract
from arXiv · showhide
A 0.6B language model, asked to verify 1,200 logical conclusions (half valid, half corrupted by a single semantic edit), answers YES every time. Judged by behavior it discriminates nothing; linear probes on its hidden states read the correct verdict at 0.96 AUC, transferring to unseen logical structures and separating foils built from exactly the words of the true conclusion (0.90). We ask where the verdict is lost, and find the dominant failure is a single scalar. The verdict survives to the model's own output logits (margin AUC 0.89) along a well-aligned readout direction; a saturated decision threshold, offset by +4.6 sigma, erases it. The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset (Spearman -0.93) while margin ranking moves far less. Across a 13x scale range, internal knowledge saturates while free-form behavior is non-monotone: an 8B model underperforms its 4B sibling through an answer-channel failure rather than the threshold; forced-choice accuracy is monotone. The diagnosis is actionable: a one-parameter correction, never fit on evaluated structures, repairs behavior from 50% to 81% (0.6B); calibrated margin decoding recovers 94% at 8B; few-shot prompting works the same way, recentering the threshold (+4.6 sigma to 0.0 sigma) while preserving ranking. Comparing probe to margin separates three regimes: concealed, miscalibrated, and undetected. On a maze task built so foils carry no surface cues, the audit correctly reports the third. In the standard generation setting, answer-surface features and heuristic labels reproduce published probing results without any internal access.
Introduction
The paper tests whether language models can internally compute correct answers while failing to express them, and traces this gap from hidden states to output behavior. Across controlled logic, scale, prompting, and maze evaluations, it finds that threshold miscalibration often explains the gap, while probe–margin comparisons distinguish miscalibration from concealment or absence.
- Motivation: 50.0% accuracy resulted from answering YES on all 1,200 logic trials, despite hidden-state probes reaching 0.96 AUC on unseen structures.The foils differed from valid conclusions by one semantic edit, making the labels exact.
- Mechanism: 0.887 margin AUC showed that the model’s own YES−NO logits ranked validity without probes or training.The readout direction was aligned with the verdict; the dominant failure was the decision threshold.
- Generalization: Behavioral accuracy across 90 semantic-label configurations correlated with threshold offset at Spearman −0.93, while margin ranking moved far less.The configurations varied models, phrasings, label vocabularies, and option orders.
- Mechanism: +4.6σ threshold offset made greedy decoding always output YES, while one fitted threshold restored behavior to 81%.The correction changed the decision boundary rather than the ranking of trials.
- Scale: At 8B, forced-choice accuracy rose monotonically while free-form accuracy fell to 84.8% because 113 of 1,200 responses contained neither answer token.Calibrated margin decoding reached 94.2%, bypassing this free-form failure.
- Diagnostic: In a maze task without candidate-only cues, the audit reported no recoverable verdict on long paths and a miniature ladder on shorter paths.The diagnostic therefore distinguishes undetected knowledge from other failures.
Related Work
The paper connects hidden-knowledge probing with output-label calibration and evaluates whether apparent internal knowledge exceeds information available from answers themselves. It argues that answer-surface controls, logit margins, and exact-label designs are necessary to distinguish representation signal from probe or labeling artifacts.
- Hidden knowledge: Prior work established that models can encode correctness, truth directions, and error estimates in internal states, motivating tests of knowledge–behavior dissociation.The cited literature includes activation probes, truth-direction recovery, and interventions on truthfulness-related heads.
- Probing methodology: Probing methodology requires controls because decodability does not establish that the model uses the decoded information.This paper treats answer text, task difficulty, frozen encoders, and the model’s own logit margin as relevant null comparisons.
- Label calibration: Prompted classification is distorted by label priors, example order, label frequency, surface competition, and option-position bias.Prior work repairs these effects through prior estimation or affine correction of output distributions.
- Contribution: The paper opens the output channel to measure how much apparent hidden knowledge is output-distribution pathology versus concealment beyond the logits.It reports that the verdict is internally near-ceiling and aligned with the readout direction.
A Controlled Testbed
The controlled testbed converts a structured logic corpus into exact one-word verification trials with held-out structures and carefully designed foils. A complementary maze task removes candidate-surface cues, while nested probing evaluates generalization without layer-selection optimism.
- Corpus and verification task: The verification corpus contains 30 natural-deduction structures across 20 topical domains, yielding 600 English instances and paired valid and foil trials.Entire structures are held out to detect structure-identity shortcuts.
- Foils: Minimal foils differ from true conclusions by exactly one semantic edit, including polarity flips, entity swaps, and conjunct swaps.Conjunct swaps preserve the words while producing a non-entailed proposition.
- Spatial verification: The maze domain uses formally indistinguishable coordinate pairs so candidate plausibility cannot reveal which endpoint is correct.An ant follows cardinal moves, and the candidate states its final position.
- Probing protocol: Nested leave-one-structure-out evaluation selects probe layers inside each training fold, preventing layer-selection optimism.Probes use regularized logistic regression on standardized single-layer, single-position states.
Knowledge, Expression, and a Diagnostic
The paper defines knowledge and expression through increasingly restrictive access levels, then uses probe–margin comparisons to diagnose where verdict information is lost. The resulting ladder formalizes a distinction between internal decodability, logit ranking, and emitted answers.
- Definitions: Knowledge is defined as linear predictability of validity from hidden states, while expression is predictability from emitted answers; both are estimated with AUC.The framework uses V-information under computational constraints.
- Readout bottleneck: Under forced-choice decoding, the answer is a deterministic function of the final-state YES−NO margin, but free-form generation may emit neither answer token.The chain from final state to margin to answer is treated as successive information-reducing stages.
- Empirical ladder: The controlled Qwen3-0.6B table reports zero behavioral information alongside structure-held-out probes that classify validity across foil families.Layer selection is nested within folds, and confidence intervals are clustered over structures.
- Diagnostic: Three regimes follow from comparing probe AUC with margin AUC: miscalibration, misalignment, and absence.High probe and high margin indicate miscalibration; high probe with low margin indicates concealment; both low indicate no additional linearly decodable verdict.
Verification: Knowledge Without Behavior
The model’s behavior is at chance, but its hidden states encode logical validity across held-out structures and difficult surface-matched foils. The signal develops internally and includes both premise-dependent verification and candidate-intrinsic plausibility.
- Behavioral collapse: 50.0% balanced accuracy results from answering YES on all 1,200 trials, while negating the question lowers accuracy to 38%.The behavior is not explained by option order: reordering leaves every answer YES.
- Internal verdict: 0.964 probe AUC and 89.8% accuracy recover validity from answer-token states despite 50.0% behavioral accuracy.All 30 held-out logical structures reach at least 0.90 probe AUC.
- Layer development: Probe AUC rises from 0.50–0.60 through layer 4 to 0.95–0.96 by layer 16, above lexical 0.76 and frozen-encoder 0.64 references.Behavior remains at chance throughout the layer profile.
- Surface controls: 0.897 AUC on foils sharing the true conclusion’s entire word multiset shows the probe detects more than surface overlap.The expanded foil macro-average is 0.900, while candidate–premise overlap reaches only 0.759 and a frozen encoder 0.642.
- What the probe reads: 0.962 matched-probe AUC exceeds 0.817 candidate-only and 0.828 premise-shuffled AUC, revealing both premise-dependent and candidate-intrinsic channels.On role swaps, the premise gap reaches +0.32; on negations, candidate-only AUC reaches 0.933.
Where the Verdict Is Lost
The verdict is usually present in the model’s output distribution, but a miscalibrated threshold converts positive margins into uniform YES answers. Across prompts and scales, threshold offset explains behavioral variation, while grounding and emission failures mark distinct boundaries.
- Threshold failure: 0.887 margin AUC sits below 0.964 probe AUC at 0.6B, while the mean margin is +4.6σ above threshold and positive on 100% of trials.The strict text < margin < probe ladder shows the dominant loss occurs at the decision threshold rather than before the output distribution.
- Readout alignment: d′ along the model readout axis is 1.72 versus 2.28 for the probe at 0.6B, but equalizes at 3.74 versus 3.74 by 4B.The readout direction is imperfect at 0.6B and aligned with the probe by 4B, making direction error secondary to threshold offset.
- Prompt variation: Spearman −0.93 links behavioral accuracy to threshold offset across 90 semantic-label configurations, with per-model 4B spread shrinking from SD 0.107 to 0.022 after correction.The result indicates that offset, rather than ranking, carries the observed prompt sensitivity in these configurations.
- Boundary condition: Calibrated accuracy is approximately 0.50 for arbitrary symbol labels when larger-model margins invert to margin AUC 0.03, identifying grounding failure rather than calibration.Bias correction cannot repair a sign-flipped binding between verdicts and arbitrary symbols.
- Scale and emission: Free-form accuracy falls from 93.2% at 4B to 84.8% at 8B despite margin AUC rising from 0.979 to 0.980, because 113 of 1,200 responses contain neither answer token.Forced-choice accuracy instead rises monotonically from 0.500 to 0.923 to 0.943 across 0.6B, 4B, and 8B.
Repair, Elicitation, and the Diagnostic
A single held-out scalar correction repairs much of the behavioral failure, and few-shot prompting achieves a similar threshold recentering while preserving ranking. Comparing margins with probes then distinguishes repairable miscalibration from concealed or undetected computation across models and tasks.
- Held-out repair: LOSO scalar calibration raises 0.6B behavior from 50.0% to 81.0% and yields 94.2% calibrated margin decoding at 8B.The scalar is fit on 29 structures and evaluated on the held-out 30th; median-centering matches the supervised correction at 81.0% and 94.7%.
- Elicitation: Few-shot prompting moves the threshold from +4.64σ to +0.01σ while changing AUC only from 0.880 to 0.893 and preserving rank correlation at 0.79.Behavior rises from 50.0% to 81.9% under first-token scoring, closely matching statistical calibration.
- Residual concealment: At 4B, a probe recovers 63.4% accuracy on the model’s 82 errors, raising overall performance to 95.0% versus 93.2% behavior.The probe–margin gap remains small but excludes zero at both 4B and 8B.
- Cross-model audit: Table 4 preserves the behavior < margin < probe ordering across models where probes were fit, while cross-family probing transfers from Qwen3-0.6B structures to Phi-3.5 states.Phi’s already-centered margin is explicitly marked as matching behavior after calibration.
- Diagnostic regimes: On the maze’s hard tier, margin 0.538, probe 0.581, and candidate-only 0.500 indicate no linearly decodable verdict beyond the logits.The audit therefore reports an undetected regime rather than manufacturing a probe–margin gap.
Generation: Why the Controls Matter
In standard generation, apparent hidden correctness can be reproduced without internal access because visible answer features and heuristic labels carry predictive structure. Adjudicated labels and transfer controls therefore materially change which predictors appear effective.
- Surface baselines: A 10-feature visible-answer baseline matches probes against heuristic labels in transfer AUC, 0.870 versus 0.856.This shows that answer-surface statistics can reproduce the apparent probing result without hidden-state access.
- Label controls: Adjudication changes strict correctness from 15.7% to 5.5%, and a four-feature difficulty baseline beats every hidden-state probe under transfer.The four features are premise count, depth, and prompt length as described in the passage.
Discussion and Conclusion
The paper argues that apparent hidden knowledge is often output-logit knowledge lost at a miscalibrated threshold, making the gap between competence and expression narrower and more repairable than assumed. It also bounds this conclusion by showing that training dynamics remain open and the evidence is limited to synthetic English tasks and models up to 8B.
- Interpretability: Probing claims should include margin and prior-corrected baselines because most apparent hidden knowledge is already present in output logits.The concealed remainder beyond log-probability access is reported as +0.008–+0.076 AUC.
- Evaluation: Text-only evaluation combines ranking with thresholding, and the threshold term dominates prompt sensitivity.Where log probabilities are available, margin AUC and a fitted threshold separate these components.
- Training story: The offset is a small-model pretraining property that scale removes and format perturbs, while its fine-tuning dynamics remain open.The base 0.6B model shows the offset, whereas the base 4B model is already calibrated.
- Limitations: The evidence is limited to synthetic English domains and models no larger than 8B, and the probes remain correlational.Whether the observed ordering holds on natural, noisy-label tasks is untested.
- Conclusion: A premise-dependent validity signal reaches the model’s logits but is lost at one miscalibrated threshold that one fitted parameter repairs.The paper concludes that the knowledge–expression gap is later and more fixable than the hidden-knowledge framing suggests.