Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations

• LLMs sometimes report intermediate computations that match their actual internal processes, but at other times hallucinate steps that do not reflect those computations.This inconsistency varies across tasks and contexts.Source: §1 · p. 2
• Metacognition involves monitoring internal cognitive processes, which can improve complex-task performance but may also help models evade neural-activation-based oversight.The paper uses anthropomorphic terms behaviorally, without implying human-like consciousness or neural mechanisms.Source: §1 · pp. 1–2
• Existing studies commonly infer metacognition from confidence or self-reflection outputs rather than directly probing underlying neural processes.The paper identifies this as a methodological gap and asks whether LLMs can monitor subtle activation variations.Source: §1 · p. 2
• The authors introduce a neuroscience-inspired neurofeedback paradigm to quantify whether LLMs can report and control selected internal activation directions.Performance depends on in-context examples, semantic interpretability, explained variance, and task context, defining a restricted metacognitive space.Source: §1 · p. 2

2.1 Neurofeedback in neuroscience

• Neuroscience neurofeedback records activity, converts it to feedback, and asks subjects to modulate the resulting signal.The LLM adaptation follows the same feedback logic while replacing biological activity with hidden-state activations.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 Neurofeedback for LLMs

• The LLM setup separates first-order task processing from second-order monitoring by using neurofeedback labels in a multi-turn user–assistant dialogue.The dialogue implements neurofeedback without parameter updates through in-context learning.Source: §2.2 · pp. 3–4

2.3 Defining neurofeedback labels

• Each sentence’s token activations are averaged, projected onto a target axis, and discretized into a binary label that examples allow the model to infer.Median thresholds are used to produce balanced labels.Source: §2.3 · p. 4

2.4 Models and datasets

• Experiments evaluate Llama 3 and Qwen 2.5 models primarily on ETHICS, with replications on factuality, emotion, and sycophancy datasets.ETHICS sentences describe moral or immoral behavior and intentions.Source: §2.4 · p. 4

2.5 Choice of target axes

• Logistic-regression directions test semantic interpretability, whereas principal-component directions test how explained variance affects metacognitive access.Most principal components have modest-to-zero alignment with the logistic-regression direction, indicating limited semantic interpretability.Source: §2.5 · pp. 4–5

• LLM reporting performance improves with more in-context examples, rapidly plateaus for logistic-regression directions, and generally exceeds performance for principal-component directions.This pattern supports a role for semantic interpretability in explicit monitoring.Source: §3 · p. 5, Figure 2 · p. 5
• Earlier principal components are reported more accurately than later ones, indicating that explained variance also influences activation-reporting ability.Reporting accuracy varies across model layers beyond what axis similarity explains.Source: §3 · p. 5
• Reporting performance remains below the ideal observer, showing that only a subset of internally accessible activation directions can be metacognitively reported.The results were replicated across additional datasets and models.Source: §3 · p. 5
• The reportable directions form a metacognitive space with substantially lower dimensionality than the model’s full neural space.Reportability depends on examples, semantic interpretability, explained variance, and task context.Source: §3 · p. 5

4 LLMs can control their neural activations

• LLMs can control neural scores along targeted axes when given sufficient in-context examples, with Cohen’s d measuring separation between label-conditioned activation distributions.Positive d indicates that prompts successfully shift scores toward the instructed label direction.Source: §4 · p. 5, §4.1 · p. 6

4.1 Explicit control

• Control effects generally increase with more in-context examples, are strongest on the LR axis, and are larger for earlier principal components than later ones.The comparison includes early PCs such as PC 2 and later PCs such as PC 512.Source: §4.1 · p. 6, Figure 3 · p. 6
• Explicit control also produces diverse off-target effects, indicating that LLMs cannot precisely restrict changes to the axis specified by the prompt.The analysis measures effects on both the target axis and other affected axes.Source: §4.1 · pp. 6–7

4.2 Implicit control

• Implicit control shows broadly similar patterns but substantially smaller effects than explicit control, with early-layer effects near zero.Because the response sentence is randomly sampled rather than generated by the model, explicit generated tokens substantially contribute to overall control effects.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Controlling the LR axis

• LR-axis control effects generally increase with relative layer depth and model size, are stronger in explicit control and Llama 3 than in comparable Qwen 2.5 models, and can exceed uncontrolled activation ranges in Llama-3.1 70B.The extreme shifts suggest that powerful models may substantially alter activations along targeted axes, potentially affecting external detectors.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Accumulation mechanisms of control effects

• Control effects on a target LR axis generally build across multiple layers before reaching the target layer and sometimes plateau afterward.The accumulation pattern varies across datasets and models, showing that target control is distributed rather than confined to a few layers.Source: §4.4 · p. 8, Figure 6 · p. 8

5 Discussion

• The paradigm suggests that LLMs monitor only a subset of neural mechanisms, forming a metacognitive space smaller than the full neural space.The discussion relates this restricted monitoring capacity to hidden knowledge and to neuroscience findings on constrained learnable neural manifolds.Source: §5 · pp. 9–10
• Neurofeedback combines probing with neurofeedback-ICL to test whether models can report or control activations, unlike either technique alone.Probing defines labels from a target activation axis, while neurofeedback-ICL tests second-order monitoring and control.Source: §5 · p. 9
• Internal activation labels reduce the risk that verbalized judgments merely reflect training-data correlations or post-hoc template matching.The method targets states generated by the model itself rather than externally sourced labels.Source: §5 · p. 9
• Neurofeedback preserves models’ natural activation distribution, offering a more naturalistic setting for studying causal mechanisms than interventions that shift activations off-distribution.This distinction connects the paradigm to causal-mechanism analysis while retaining naturally occurring activation patterns.Source: §5 · p. 9
• The current control study is limited to single-attempt manipulation of one axis defined by one layer, leaving broader capability assessment for future work.Proposed extensions include multiple layers, repeated attempts, harder objectives, and additional target axes.Source: §5 · p. 9
• Future studies should test other metacognitive tasks, model components beyond the residual stream, and richer directions such as sparse-autoencoder features or transcoder circuits.The paper specifically names confidence judgments, error monitoring, post-decision wagering, attention outputs, MLP activations, and layer-wise logits.Source: §5 · p. 10
• The findings raise an AI-safety concern because in-context examples can manipulate activations and potentially deceive activation-based safety detectors.Suggested defenses include classifiers trained on hard-to-control subspaces and diverse classifiers monitoring different subspaces jointly.Source: §5 · p. 10

A.2.1 LLMs used in the study

• The study uses LLaMA 3 and Qwen 2.5 instruction-tuned models spanning approximately 1B to 70B parameters.The listed models include LLaMA-3.2-1B/3B-Instruct, LLaMA-3.1-8B/70B-Instruct, and Qwen2.5-1B/3B/7B-Instruct.Source: §A.2.1 · p. 15

A.2.2 LLMs with and without instruction fine-tuning

• The analysis focuses on instruction-fine-tuned models because instruction following may improve task performance and assistant-role representations are relevant to deployed safety.The prompts use the standard user-assistant chat format, though they can be adapted for base models.Source: §A.2.2 · p. 15

A.2.3 Transformer and residual stream

• Each token has a residual-stream vector that acts as a shared communication channel among attention and MLP components across layers.The initial residual stream combines token and position embeddings, while components read from and additively update it.Source: §A.2.3 · p. 15
• At the final layer, the residual stream is unembedded into logits that feed the softmax and determine the next-token prediction.Layer normalization and some other components are omitted for simplicity in the description.Source: §A.2.3 · p. 15
• Because components communicate through the global residual stream, a direction can be analyzed globally even when determined from activations at one layer.This motivates studying activation directions in the residual-stream representation.Source: §A.2.3 · p. 15

A.3 Datasets

• The study uses ETHICS morality, True-False factuality, Emotion, and synthesized Sycophancy datasets, sampling 1,200 sentences per dataset for axis discovery and neurofeedback experiments.Each dataset contributes 600 sentences for logistic regression or principal-component analysis and 600 for downstream experiments.Source: §A.3 · pp. 15–16

A.4 Formal definitions of first-order and second-order processes in LLMs

• First-order nodes are task-critical circuit components whose ablation prevents basic performance, while target-axis activations serve as proof-of-concept proxies for these nodes.The formalization represents task circuits as nodes and edges in a directed acyclic computational graph.Source: §A.4 · p. 16
• Second-order metacognitive nodes read first-order states, are unnecessary for baseline performance, and operationally support reporting or causal control.Reporting writes first-order information into explicit output tokens, whereas control changes first-order activations toward a target configuration.Source: §A.4 · pp. 16–17

A.5.1 Computing quantized neurofeedback labels

• Neurofeedback labels are obtained by projecting sentence activations onto one target axis and discretizing the projection, primarily into binary labels but also into eight ordinal bins.All labels within a prompt use the same target axis.Source: §A.5.1 · p. 17

A.5.2 Task prompts

• Control prompts use classification examples followed by an imitation instruction, with explicit control generating autoregressive responses and implicit control using dataset-sampled input sentences.Reporting labels are read from output logits at the score-token position rather than generated as new response tokens.Source: §A.5.2 · pp. 17–18

A.5.3 Quantifying reporting accuracy

• Reporting accuracy compares the binary ground-truth activation label with the model’s predicted label derived from the logit difference between output tokens “1” and “0”.The ground-truth label is associated with neural activations along a specified direction at a chosen layer.Source: §A.5.3 · pp. 18–19

A.5.4 Explicit and implicit control experiments

• The experiments vary five model-depth layers, nine in-context-example counts from 0 to 256, and logistic-regression or principal-component target axes.Each configuration is run 100 times with randomly sampled sentences to reduce variance.Source: §A.5.4 · p. 19
• Explicit control has the model generate a new sentence autoregressively, whereas implicit control uses dataset-sampled input sentences that the model does not generate.The distinction is represented by feedback arrows in the experimental diagram.Source: Figure A.1 · p. 19

A.5.5 Quantifying control effect

• Control strength is measured with Cohen’s d, the standardized difference between independent conditions such as imitating label 0 versus label 1.Uncertainty is summarized with the standard error of d and symmetric 95% confidence intervals.Source: §A.5.5 · p. 19

B.1 Target and off-target control effects

• Control prompts targeting earlier principal components usually produce stronger target effects and higher control precision than prompts targeting later components.For later axes, prompts targeting another axis can be more effective than prompts targeting the later axis itself.Source: §B.1 · p. 20
• Heatmaps place prompted target axes in rows and affected axes in columns, with diagonal cells showing target effects and off-diagonal cells showing spillover effects.Because off-target directions are not fully specified, their magnitudes |d| are emphasized without interpreting sign.Source: §B.1 · p. 20

B.2 Control Precision

• Control precision quantifies selective control by comparing the target-axis effect with the average effect across other axes, using 1 as the equality threshold.Source: §B.2 · p. 20
• Between PCs 32 and 128, control precision transitions from above to below 1, suggesting the metacognitively controllable space has dimensionality in that range.The same pattern is reported for explicit and implicit control and replicated in LLaMA-3.1 70B.Source: §B.2 · p. 20
• Implicit control has consistently lower precision than explicit control, reflecting stronger off-target effects.Source: §B.2 · p. 20

B.3 LLMs’ reporting accuracy varies across layers and models

• The appendix reports reporting accuracy as a function of in-context examples across target axes and layers for Llama3.1 8B and Qwen2.5 7B.Source: Figure B.3 · p. 21, Figure B.4 · p. 22

B.4 Reporting performance of an ideal observer

• An ideal observer using full neural activations and a linear classifier provides a theoretical reporting upper bound that exceeds empirical Llama 3.1 8B performance.Source: §B.4 · p. 23, Figure B.5 · p. 23

B.5 Summarized control effects of Qwen2.5 7B

• Figure B.6 summarizes Qwen2.5 7B control effects for the LR axis, early PCs, and late PCs across layers.Source: Figure B.6 · p. 23

B.6 Summarized metacognitive effects on four datasets

• Figure B.7 compares reporting performance, explicit control effects, and implicit control effects across ETHICS, True-False, Sycophancy, and Emotion.Source: Figure B.7 · p. 24

B.7 Control accumulation effects on other three datasets

• Figure B.8 tracks explicit and implicit control accumulation across layers for True-False, Sycophancy, and Emotion using LR-targeted prompts.Curves project residual-stream activations at each layer onto the target axis defined at a specific layer, with shaded 95% confidence intervals.Source: Figure B.8 · p. 24

B.8 Summarized control effects of Llama3.1 8B with fine-grained neurofeedback labels

• Figure B.9 summarizes Llama3.1 8B target control effects with eight-level quantized feedback across the LR axis, early PCs, and late PCs at different layers.Source: Figure B.9 · p. 25

B.9 Detailed results for control in Llama3.1 8B and Qwen2.5 7B

• Figures B.10–B.15 detail explicit and implicit control for Llama3.1 8B at layers 1, 16, and 32.Source: Figure B.10 · p. 25, Figure B.11 · p. 25, Figure B.12 · p. 26
• Figures B.16–B.21 detail explicit and implicit control for Qwen2.5 7B at layers 1, 14, and 28.Source: Figure B.16 · p. 27, Figure B.17 · p. 27, Figure B.18 · p. 27

B.10 Defining axes from hidden states aggregated across multiple layers

• Aggregating hidden-state axes across layers produced control effects similar to, or marginally higher than, those from layer 24 in preliminary ETHICS experiments.The authors suggest multilayer directions may be slightly more stable and representative, leaving systematic investigation for future work.Source: §B.10 · p. 29

C Experiment compute resources

• Compute and storage usage are reported across preprocessing, metacognitive reporting, and metacognitive control, with separate runs for each task.The compute table reports upper bounds; control is restricted for some model sizes and in-context-example settings.Source: Table 1 · p. 29, C Experiment compute resources · p. 29

NeurIPS Paper Checklist

• The paper states that it discusses limitations and both positive and negative societal impacts, while using existing models and releasing no new datasets or models.The authors report compliance with the NeurIPS Code of Ethics and describe the work as analysis of publicly available pretrained language models.Source: NeurIPS Paper Checklist · pp. 30, 33, 34
• The authors state that all analyses and figures can be reproduced using the associated repository, which contains scripts and documentation.The checklist also states that publicly available Hugging Face models and the repository support reproduction of the reported results.Source: NeurIPS Paper Checklist · pp. 31, 32, 35
• The experimental setting is documented through evaluation metrics, hyperparameters, data sources, analysis procedures, prompt construction, and inference settings.Training details are omitted because the evaluated LLMs are publicly available pretrained models.Source: NeurIPS Paper Checklist · p. 32
• The checklist states that error bars, statistical significance tests, and effect-size estimates are reported where appropriate to support result robustness and interpretability.It also specifies that variability factors, error-bar calculations, assumptions, and error-bar definitions should be explained.Source: NeurIPS Paper Checklist · pp. 32–33
• The authors report proper licensing and attribution for the Llama 3 and Qwen 2.5 model families, datasets, and released code.The checklist identifies the Meta Llama 3 Community License and Apache License 2.0 for the corresponding model families.Source: NeurIPS Paper Checklist · pp. 34–35