Source-linked AI summary
Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits
Amirhosein Ghasemabadi, Di Niu
TL;DR
LLMs often fail to recognize their own errors, while existing self-evaluation approaches rely on text, multiple samples, or external judges. Gnosis reads hidden-state and attention traces from frozen models to predict correctness, outperforming large judges across benchmark domains with a small, low-overhead head. Its scope is sibling-model self-awareness, not universal zero-shot judging across unrelated architectures or generation styles.
Problem
LLMs can produce fluent, confident errors yet remain unreliable at assessing their own outputs, while common evaluation approaches use external judges, multiple samples, or text-based critique.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Gnosis shows that correctness cues are intrinsic to generation and can support compute-efficient systems that detect failing trajectories with negligible overhead.
Source: §7 · p. 9Gnosis is not a universal zero-shot judge, particularly across unrelated architectures or differing generation styles.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) generate fluent and complex outputs but often fail to recognize their own mistakes and hallucinations. Existing approaches typically rely on external judges, multi-sample consistency, or text-based self-critique, which incur additional compute or correlate weakly with true correctness. We ask: can LLMs predict their own failures by inspecting internal states during inference? We introduce Gnosis, a lightweight self-awareness mechanism that enables frozen LLMs to perform intrinsic self-verification by decoding signals from hidden states and attention patterns. Gnosis passively observes internal traces, compresses them into fixed-budget descriptors, and predicts correctness with negligible inference cost, adding only ~5M parameters and operating independently of sequence length. Across math reasoning, open-domain question answering, and academic knowledge benchmarks, and over frozen backbones ranging from 1.7B to 20B parameters, Gnosis consistently outperforms strong internal baselines and large external judges in both accuracy and calibration. Moreover, it generalizes zero-shot to partial generations, enabling early detection of failing trajectories and compute-aware control. These results show that reliable correctness cues are intrinsic to generation process and can be extracted efficiently without external supervision.
1 Introduction
LLMs can produce confident errors while remaining unreliable at assessing their own outputs, and prior approaches rely on signals external to their internal generation dynamics. Gnosis reads hidden-state and attention traces to predict correctness, outperforming large external judges with a small, low-overhead head and supporting early failure detection.
1 Introduction
- LLMs often fail to recognize their own confident errors, limiting reliability, safety, and efficiency in settings involving long-horizon reasoning or compute-aware control.Source: §1 · p. 1
- Gnosis extracts reliability cues from frozen LLMs’ internal dynamics to produce intrinsic, well-calibrated correctness estimates without external judges.It observes hidden states and attention patterns during inference and adds negligible computational overhead.Source: §1 · pp. 1–2
- With ∼5M added parameters, Gnosis outperforms billion-parameter reward models and proprietary judges across math reasoning, open-domain QA, and academic benchmarks.Figure 1 reports AUROC comparisons against 8B Skywork reward models and Gemini 2.5 Pro across the three benchmark domains.Source: Figure 1 · p. 2, §1 · p. 2
- Its fixed-budget, dual-stream architecture models hidden-state evolution and attention-routing patterns across an entire generation trajectory.The descriptors capture full spatiotemporal structure and operate independently of sequence length.Source: §1 · p. 2
- A head trained on a small backbone transfers zero-shot to larger variants and predicts failures from partial generations, supporting early termination and compute-aware control.Source: §1 · p. 2
2 Related Work
Prior work estimates correctness from generated text, multiple sampled answers, external judges, or internal model signals. These approaches face trade-offs in inference cost, robustness, or accuracy; internal indicators and probes often miss the full trajectory’s spatiotemporal structure.
2 Related Work
- External reward models provide response-quality and correctness signals but require costly annotation and add serving-time latency and deployment overhead.Systems such as HelpSteer2 and Skywork-Reward-V2 emphasize large, curated preference datasets.Source: §2 · pp. 2–3
- Text-based confidence and self-critique can track fluency more than reasoning validity, require extra passes, and remain brittle across domains and sequence lengths.Training-free uncertainty proxies may also struggle with confident hallucinations and out-of-distribution shifts.Source: §2 · p. 3
Internal signal-based indicators and linear
- Internal indicators use hidden activations, attention statistics, trajectory features, or shallow probes, yet prior approaches report low accuracy across diverse benchmarks.The cited account attributes this shortfall to fragile heuristics and single-token snapshots that miss the generation’s full spatiotemporal structure.Source: Internal signal-based indicators and… · p. 3
2 Related Work
- Multi-sample self-consistency can improve robustness, but its inference cost grows with the number of samples and may saturate on long, compositional tasks.Cost-aware test-time scaling instead uses internal signals to prune search or adapt compute.Source: Internal signal-based indicators and… · p. 3
3 The Gnosis Mechanism
Gnosis passively extracts reliability signals from frozen LLMs’ internal traces and predicts correctness with fixed-budget computation. It encodes hidden-state trajectories and attention patterns separately, then fuses their descriptors into a correctness estimate.
3 The Gnosis Mechanism
- Gnosis passively compresses a frozen LLM’s evolving hidden states and attention patterns into compact descriptors for predicting correctness.Its inference cost is designed to remain independent of sequence length.Source: §3 · p. 3
3.1 Problem Setup and Length-Invariant Inputs
- Fixed-budget projection maps variable-length traces to fixed-size tensors, so downstream computation does not grow with sequence length.The backbone remains frozen throughout.Source: §3.1 · pp. 3–4
3.2 Hidden-State Circuit Encoder
- The hidden-state encoder treats the projected trace as a temporal signal, then uses Set Attention Blocks and pooling to form a global descriptor.Multi-scale depthwise convolutions capture local dependencies and irregularities in the hidden trajectory.Source: §3.2 · pp. 4–5
3.3 Attention Circuit Encoder
- Attention maps are summarized with CNN and distribution-statistics features, arranged by layer and head, then mixed with axial convolutions and pooled.The hybrid feature extractor is adopted because it is most consistent across benchmarks.Source: §3.3 · p. 5
3.4 Gated Fusion and Correctness Prediction
- A gated MLP combines hidden-state and attention descriptors into a correctness probability, adaptively weighting the two feature types per example.Gnosis adds approximately 5M parameters.Source: §3.4 · p. 6
3.5 Training
- Training uses generated answers labeled against ground-truth answers and binary cross-entropy, while gradients update only Gnosis and the backbone stays frozen.The training examples form a binary classification dataset.Source: §3.5 · p. 6
4 Experiments and Results
Gnosis is evaluated for self-judgment, sibling-model judgment, and early correctness prediction across math, open-domain QA, and academic knowledge tasks. It outperforms internal baselines and external judges, transfers across related model sizes, and can assess partial generations.
4.1 Experimental Setup
- Evaluations cover self-judgment, sibling-model judgment, and partial-generation prediction, using five frozen backbones across math reasoning, open-domain QA, and academic knowledge reasoning.The training corpus mixes math and trivia; each correctness head is trained per backbone.Source: §4 · p. 6, §4.1 · pp. 6–7
- Feature visualizations show the clearest correct–wrong separation for hidden-state features, weaker separation for attention features, and the sharpest discrimination when merged.Figure 4 presents dimensionality-reduced embeddings with KDE contours and marginal densities.Source: Figure 4 · p. 7
4.2 Self-Judgment
- Across tested domains, Gnosis outperforms training-free baselines and large external judges; across Math Reasoning and Open-Domain QA, it raises AUROC from Logit Entropy’s mid-0.7s to 0.95–0.96 and beats MLP-Prob by 7–18 AUROC points.It matches or exceeds Skywork 8B Reward Models (∼1000× larger) and Gemini 2.5 Pro.Source: §4.2 · p. 8
4.3 Cross-Scale: Zero-Shot Reward Modeling
- A 1.7B-trained head transfers to 4B and 8B siblings, reaching 0.93 AUROC on Math Reasoning versus 0.96 for a self-trained head and beating Skywork 8B across tested backbones.Transfer works best when generation styles and formatting align.Source: §4.3 · p. 8
4.4 Early Error Detection
- After 40% of a generation, Gnosis reaches near-peak accuracy and positive BSS on Math and TriviaQA; on Math it matches other methods’ full-solution performance.This generalizes zero-shot from complete-trajectory training and supports early termination or escalation of failing paths.Source: Figure 3 · p. 7, §4.4 · p. 9
5 Ablations and Analysis
The ablations show that hidden-state and attention signals contribute differently by task, with their combination strongest overall. Among attention-map extractors, CNN+Stats is adopted because it is the most consistent across domains.
5 Ablations and Analysis
- Fusion yields the best overall performance: hidden states dominate on TriviaQA, while attention contributes more on Math Reasoning and MMLU-Pro.The analysis attributes this pattern to complementary structural cues that emerge in longer reasoning.Source: §5 · p. 9
- CNN+Stats is adopted for the final model after the three attention-map variants perform similarly on Math, CNN-based variants strengthen TriviaQA, and CNN+Stats is most consistent on MMLU-Pro.The hybrid is reported as the most consistent across domains.Source: §5 · p. 9
- Further targeted ablations examine attention-stream design choices and hidden-stream encoder architecture and sizing, including comparison with a pooled-MLP baseline.The studies cover grid mixing, identity embeddings, pooling, layer selection, map downsampling, and local–global encoder design.Source: §5 · p. 9
6 Discussion and Limitations
Gnosis supports efficient self-evaluation and sibling modeling, but is not a universal zero-shot judge across unrelated model architectures or generation styles.
6 Discussion and Limitations
- Gnosis is designed for self-awareness rather than general-purpose reward modeling, limiting zero-shot judging across unrelated architectures or styles such as Thinking and Instruct models.The passage says it transfers robustly to sibling models, not universally.Source: §6 · p. 9
7 Conclusion
The conclusion presents Gnosis as a lightweight way for frozen LLMs to detect their own errors from internal traces. It reports strong comparisons with large reward models and proprietary judges, with negligible overhead.
7 Conclusion
- Gnosis interprets frozen LLMs’ hidden and attention traces to detect their errors without relying on external judges.The conclusion reports that it adds only ∼5M parameters while outperforming billion-parameter reward models and proprietary models such as Gemini 2.5 Pro.Source: §7 · p. 9
A Architecture Overview
Gnosis uses fixed-size summaries of hidden states and attention maps, then combines them to predict correctness at effectively constant sequence-length cost. Its two streams are complementary in benchmark ablations.
A Architecture Overview
- The overview’s two parallel streams process frozen-backbone traces; their fixed-size summaries yield a scalar correctness probability and allow partial-chain queries at effectively constant sequence-length cost.The output is produced by concatenating the descriptors, applying a gated MLP, then a sigmoid.Source: A Architecture Overview · p. 12, §A.2 · p. 13
A.1 Hidden Circuit Encoder.
- The hidden circuit pools final-layer hidden states to a fixed number of tokens, then uses dilated convolutions, SE blocks, and a set encoder to produce a compact descriptor.Only final-layer hidden states are used to avoid storing intermediate states for every token.Source: §A.1 · p. 12
A.2 Attention Circuit Encoder.
- The attention circuit downsamples attention maps to a fixed grid, extracts per-head descriptors, models layer–head dependencies, and aggregates them into an attention descriptor.Per-map extraction can use a CNN, predefined statistics, or their hybrid; CNN+Stats is the default because the hybrid is most consistent across domains.Source: §A.2 · pp. 12–13
- The statistics branch summarizes attention-map dispersion, spectral texture, diagonal locality, and the location and spread of attention mass.It uses predefined descriptors, including map, row, and column entropy and diagonal-band energies.Source: §A.2 · p. 13
B.1 Hidden vs. Attention Circuits
- On Math Reasoning, both single-stream variants achieve 0.92 AUROC, while fusion improves to 0.95; fusion also achieves the best overall performance on TriviaQA and matches the best result on MMLU-Pro.Source: §B.1 · p. 13
B.2 Attention Circuit Encoder Ablations
The attention circuit combines per-map feature extraction, mixing across the layer/head grid, and learned aggregation. Ablations favor complementary CNN and statistics features, lightweight axial mixing with layer/head identity, and query-based pooling.
Feature Input Representation
- CNN-plus-statistics features are the most consistent overall across benchmarks, while predefined statistics match the CNN at 0.92 AUROC on Math Reasoning and the hybrid reaches 0.80 AUROC on MMLU-Pro.The final model adopts the CNN+Stats design.Source: §B.2 · pp. 13–14, Table 6 · p. 14
Grid Topology and Layer/Head Identity
- Removing grid mixing or learned layer/head embeddings reduces performance, while a Global Transformer uses four times the parameters without improving AUROC over lightweight Axial Convolutions.These ablations indicate that attention heads interact and that their positions within the model matter.Source: §B.2 · p. 14, Table 7 · p. 14
Aggregation Strategy
- Replacing query-based PMA with mean pooling sharply reduces accuracy, favoring learned reliability prototypes over uniform averaging of attention circuits.The proposed configuration is selected for its balance of accuracy and parameter efficiency.Source: §B.2 · pp. 13–14, Table 7 · p. 14
B.3 Hidden-State Circuit Encoder Ablations
The hidden-state encoder uses a 192-dimensional design with local temporal processing followed by global set aggregation. Ablations favor both stages over simpler alternatives, and evaluation spans math, trivia QA, and out-of-distribution academic knowledge.
B.3 Hidden-State Circuit Encoder Ablations
- Reducing the hidden-state dimensions from 192 to 96 drops AUROC by 0.04, while scaling to 384 triples parameters without accuracy gain; the design therefore fixes dimensions at 192.The final design is selected for its trade-off between reliability estimation and computational efficiency.Source: §B.3 · pp. 14–15, Table 10 · p. 15
- Removing local temporal processing reduces AUROC by 0.04, and removing Squeeze-and-Excite gating or multi-scale dilation also degrades performance.The ablations support local denoising, multi-scale temporal signals, and dynamic feature reweighting.Source: §B.3 · p. 15
- Mean pooling or removing global self-attention hurts performance, while the SAB+PMA design outperforms a pooled MLP baseline.Table 9 reports lower AUROC when Phase 1 is removed or global aggregation is weakened, and substantially worse performance for the pooled MLP.Source: §B.3 · p. 15, Table 9 · p. 15
C.1 Benchmarks (extended).
- Evaluation covers competition-style math, an 18,000-question held-out trivia subset, and MMLU-Pro across 14 domains as an out-of-distribution academic knowledge benchmark.The domains test multi-step reasoning, hallucination detection and knowledge grounding, and generalization beyond the training mix, respectively.Source: §C.1 · p. 15
C.2 Metrics (extended).
Correctness prediction is evaluated on valid, parsable answers using ranking, error-detection, and probability-calibration metrics. Backbone outcome statistics provide context, while no-answer cases are excluded from the correctness-prediction evaluation.
C.2 Metrics (extended).
- AUROC ranks correct completions above incorrect ones, while AUPR-c measures recovery of correct answers and AUPR-e measures detection of errors or hallucinations.AUPR-e is noted as especially relevant under class imbalance.Source: §C.2 · pp. 15–16
- BSS above 0 indicates improvement over a prevalence baseline, while lower ECE indicates better alignment between predicted correctness probabilities and empirical accuracy.Source: §C.2 · p. 16
D Backbone Outcome Statistics
- Backbone outcome statistics report correct, hallucinated, and no-answer rates, but no-answer cases are filtered out and Table 1 metrics use only valid, parsable answers.The outcome breakdown contextualizes domain-dependent shifts in precision-recall and calibration metrics.Source: D Backbone Outcome Statistics · p. 16, Table 11 · p. 16
E Gemini As judge.
The Gemini 2.5 Pro baseline judges each question-answer pair by producing a correctness analysis and a score. Its API evaluation cost approximately $500 across the three benchmarks, and Figure 6 shows the system prompt.
E Gemini As judge.
- Gemini 2.5 Pro generates a brief correctness analysis and a scalar score from 0 to 1, parsed from required score tags.Source: E Gemini As judge · p. 16
- Evaluating Gemini through the API on the three benchmarks cost approximately $500 in total.Source: E Gemini As judge · p. 16
- Figure 6 presents the system prompt used for the LLM-as-judge baseline.Source: Figure 6 · p. 17