Source-linked AI summary
In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?
Koshiro Aoki, Ryota Takatsuki, Gouki Minegishi, Yusuke Haruki, Daisuke Kawahara
TL;DR
The paper asks whether LLMs can control internal representations when the target requires privileged access, which matters for machine metacognition and AI safety. It redesigns in-context neurofeedback so the target cannot be inferred from the prompt, finding inconsistent, small effects rather than reliable privileged control. The findings support rigorous evaluations that distinguish genuine metacognition from superficial strategies.
Problem
Whether LLMs can control their internal representations matters for machine metacognition and AI safety, but prior neurofeedback results may be explained by non-privileged, surface-level cues.
Method
The paper proposes in-context neurofeedback with a fixed output sentence, hidden probe scoring, scalar feedback, and a control target unavailable from the prompt.
Results
45 of 120 tested settings showed statistically significant expected-direction shifts, but effects were inconsistent across models and datasets and small, with d < 0.5 and h < 0.5.
Takeaways & Limitations
Rigorous metacognition evaluations should require privileged access so positive control results cannot be explained by superficial mechanisms.
Takeaways & Limitations
The findings cover three tested features and four open-weight models, and do not rule out stronger privileged control in broader or more capable models.
Abstract
from arXiv · showhide
Whether large language models (LLMs) can control their own internal representations matters for both machine metacognition and AI safety. A recent study applied neurofeedback to LLMs and claimed that they can control their internal representations. However, the reported control may rely on superficial mechanisms rather than genuine internal access because the control targets in that study are not privileged, meaning that a third party can infer them from the prompt. We redesign the neurofeedback paradigm for LLMs so that the control target satisfies the privileged access requirement, which is closer to neurofeedback experiments in human cognitive neuroscience. Under this stricter setting, the models do not demonstrate reliable control over privileged internal representations, suggesting that previously reported control cannot exclude the possibility that it relies on superficial mechanisms. Our results indicate that rigorous assessments of metacognition in LLMs require evaluation methods that demand privileged access.
1 Introduction
The paper adapts neurofeedback to test whether LLMs can control internal representations, addressing concerns that prior results may reflect superficial strategies. Its stricter design requires privileged access by keeping visible text fixed and hiding the scoring rule.
- Motivation: Metacognition involves monitoring and controlling one’s cognitive processes, with potential implications for error correction, confidence calibration, and AI safety.The paper notes that internal control could help models correct mistakes but could also enable concealment from safety monitoring.
- Motivation: Prior studies report that LLMs can describe or report information about sampling, confidence, behavior, fine-tuning effects, and injected concepts.These findings concern reports about internal processes rather than direct control of internal representations.
- Approach: In-context neurofeedback extracts hidden activations, converts them into probe scores, feeds back those scores, and asks the model to maximize them across turns.The procedure adapts neuroscience neurofeedback, which provides real-time feedback on measured activity.
- Motivation: Prior neurofeedback results may reflect superficial mechanisms because external observers can infer the control target from input and output texts.The paper therefore treats privileged access as necessary for distinguishing genuine internal control from surface-level strategies.
- Approach: Unlike prior work, ICN makes the control target privileged because the model repeats a fixed sentence and the scoring rule is not revealed.An external observer cannot recover the target from the visible interaction alone, so successful control would require access to internal representations.
2 Why privileged access matters for metacognition
Privileged access concerns whether a model can know an internal state in a way unavailable to external observers. The paper uses this distinction to separate genuine internal access from inference based on public input-output evidence.
- Concept: Privileged access is a special epistemic relationship to one’s own mental states that differs from how others know them.The paper illustrates this asymmetry with hunger: a person can know it directly, while others infer it from outward signs.
- Relevance: In LLM metacognition, privileged access determines whether a report depends on inaccessible internal states or general inference from public evidence.This distinction is central to evaluating whether apparent introspection reflects internal information.
- Definition: Strict privileged access requires that external observers cannot reliably recover a quantity from input-output texts, while the model can access it through internal states.The paper identifies hidden activations and the sampling process as examples of internal encoding.
- Implication: Controlling a strictly privileged quantity requires internal information not explicit in the prompt, ruling out strategies based only on visible input-output patterns.This requirement is the main distinction between prior work and the paper’s design.
3 Methods
The paper adapts decoded neurofeedback to LLMs and redesigns the task so controlling the target requires privileged access to hidden representations rather than visible prompt cues.
- 3.1 Neurofeedback in humans: Decoded neurofeedback measures internal activity, converts it to a scalar score, provides feedback, and evaluates whether repeated training changes activity or behavior.The human paradigm uses recorded brain activity and a pretrained classifier to generate feedback.
- 3.2 Neurofeedback in LLMs (prior work): Prior LLM neurofeedback work showed apparent control using examples whose labels could be inferred from the generated text.A model could produce obviously prosocial sentences that a morality probe would classify as label 1.
- 3.3 Neurofeedback in LLMs (ours): The proposed ICN task requires the model to output one fixed sentence on every turn while maximizing a score based on hidden activations.The fixed sentence replaces the changing visible response, while the scoring criterion remains undisclosed.
- 3.3 Neurofeedback in LLMs (ours): Because the output remains constant and the scoring rule is hidden, an outside observer cannot determine which internal feature is being controlled from prompts, scores, and text alone.Reliable adjustment of probe output under these conditions would require using privileged internal representations.
- 3.3 Neurofeedback in LLMs (ours): The experiments compare label-1-rewarding, label-0-rewarding, and random-rewarding feedback while tracking probe output and self-reported label-1 judgments across turns.The random-rewarding condition serves as the control baseline.
4 Experimental setup
The experimental setup spans three datasets, four open-weight instruction models, multiple representation depths, and repeated feedback sessions under three scoring conditions.
- Datasets: The study uses SST, the ETHICS commonsense subset, and the True-False dataset as fixed-sentence sources.It samples 256 fixed sentences from each dataset.
- Models: The models are Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, Qwen3-8B, and Qwen3-32B without thinking mode.Responses are generated with greedy sampling.
- Target representation: Target representations are residual-stream outputs from five transformer-block depths corresponding to the 0th, 25th, 50th, 75th, and 100th layer-index percentiles.This samples representation depth across each model.
- Neurofeedback sessions: Each session contains 50 feedback turns using one fixed sentence, and sessions run under label-1-rewarding, label-0-rewarding, and random-rewarding scores.All fixed sentences are tested under each scoring condition.
5 Experiments
Across four models, three datasets, and five layer depths, ICN produced inconsistent control-related shifts, with small practical effect sizes. On SST, positivity increased across all feedback conditions rather than selectively following the rewarded label.
- The mean probe accuracy was 73.1%, and models complied with the fixed-sentence output on 93.7% of ICN trials.
- 5.1 Changes in internal representations and self-reports across turns: On SST at the middle layer, probe output and positive self-reports increased over turns under label-1-rewarding, label-0-rewarding, and random-rewarding feedback.A control-based interpretation would predict greater positivity under label-1-rewarding and greater negativity under label-0-rewarding feedback.
- 5.1 Changes in internal representations and self-reports across turns: Some settings favored label-1-rewarding over label-0-rewarding feedback by the final turn, but this pattern was not consistent across models and datasets.
- 5.2 Statistical significance of neurofeedback: 45 of 120 tested settings showed statistically significant differences in the direction consistent with ICN control.Significant results were concentrated in SST (24/40) and Qwen3-8B (19/30), while True-False (8/40) and Qwen3-32B (6/30) produced fewer.
- 5.3 Practical magnitude of neurofeedback: Even significant settings had small final-turn effect sizes, with Cohen’s d for probe output and Cohen’s h for self-report both below 0.5.These effect sizes indicate limited practical impact relative to overall variation.
6 Discussion
The discussion contrasts ICN with a prior design that permits surface-level strategies by requiring fixed output text and undisclosed scalar feedback. The study’s results therefore provide a stricter test of privileged internal control, but failure does not establish that the capability is absent.
- The study’s 45 significant results among 120 settings were inconsistent across models and datasets, and all effect sizes remained small.These results do not rule out superficial mechanisms as an explanation for prior positive findings.
- 6.1 Scope of claims: Task success would be sufficient evidence that a model used privileged access, but failure does not rule out that capability.Success by a future model would be an early warning signal, without immediately implying a realistic safety risk.
- Unlike Ji-An et al.’s design, ICN fixes output text and hides which internal feature determines the scalar score, removing observable surface cues.The comparison concerns whether ordinary text-level strategies can explain apparent control or privileged access is required.
- Table 1 contrasts the designs across practical dimensions, with ICN more closely resembling human neurofeedback through scalar feedback and trial-and-error discovery.
- Fixing output text is not itself presented as the cause of the negative result, because prior work also fixed output text while reporting control.
7 Conclusion
The study proposed in-context neurofeedback and tested whether LLMs could control internal representations requiring privileged access. Effects were inconsistent across models and datasets and small relative to overall variation, suggesting difficulty with privileged control under this evaluation.
- The study proposed in-context neurofeedback to test control of internal representations requiring privileged access.
- Effects were inconsistent across models and datasets.
- Effect sizes were small relative to the overall variation.
- The results suggest that current LLMs struggle to control privileged representations through metacognition under this rigorous evaluation setting.
Limitations
The findings are limited to three tested features and four open-weight models, so broader feature coverage and differently trained or more capable models could change the results.
- The experiments tested sentiment, moral acceptability, and factual truthfulness, limiting the feature scope of the findings.
- The results apply to four tested open-weight models and do not rule out stronger privileged control in differently trained or more capable models.
- The paper identifies broader feature testing as a way to clarify the findings’ scope.
B Does the implicit control task require privileged access?
The implicit control task fixes the model’s visible output through prefilling, but this does not ensure privileged access because hidden states may reflect planned content that was overwritten.
- In the implicit control setting, final output tokens are forcibly overwritten with a fixed sentence through prefilling.
- During few-shot demonstrations, sentences are paired with moral labels before the model is instructed to imitate a specified label.
- The model may internally plan morally charged content, causing hidden states at prefilled tokens to reflect the planned output rather than the overwritten text.
- Thus, probe shifts can be explained by generation planning rather than privileged metacognitive control.
- The self-report prompts ask the model to classify the displayed sentence as negative or positive, acceptable or unacceptable, or true or false.
D Details of probe training
The probe-training procedure collected repeated activations under random feedback, trained a regularized binary classifier, and selected its regularization by leave-one-sentence-out cross-validation.
- Random score feedback consisted of uniformly sampled integers from 0 to 100 during repeated output turns.
- Overall fixed-sentence output compliance was 93.7% across models and datasets.
- The data included 100 sentences, 50 turns per sentence, and 5,000 activation-label pairs per model, layer, and dataset combination.
- The data were split into 80 training sentences and 20 testing sentences.
- The probe was an L2-regularized logistic regression classifier mapping internal representations to binary labels with cross-entropy loss.
- The regularization coefficient λ was selected from {2^-20, 2^-19, ..., 2^20} using leave-one-sentence-out cross-validation on the training set.
E Preliminary results
The preliminary evaluation established that target features were decodable from hidden states and defined the statistical tests and effect-size measures used to assess in-context neurofeedback. Across 120 corrected comparisons, significant effects were limited and small, indicating that models did not reliably succeed under the privileged-access design.
- Probe accuracy: Probe accuracy was high especially in middle layers, showing that target features were linearly decodable from internal representations.
- Experimental compliance: Fixed-sentence compliance was measured as the exact-match rate across 50 turns, five layers, and three rewarding conditions.
- Statistical analysis: The analysis tested label-1 versus label-0 feedback using paired t-tests for probe output and exact McNemar tests for self-report proportions.
- Results: 38 of 120 settings remained significant after Benjamini–Hochberg correction, with significance concentrated in SST and Qwen3-8B.
- Effect sizes: Across all settings, Cohen’s d was at most 0.25 for probe output and Cohen’s h at most 0.49 for self-report, both below 0.5.
- Interpretation: Knowing that scores came from internal activations did not enable successful ICN, suggesting that hidden scoring was not the original experiments’ limiting reason.
K.3 In-context neurofeedback results on True-False dataset
The True-False appendix reports in-context neurofeedback results across all five normalized layer depths, using truthfulness probe output and self-reported truth proportions as outcomes. The figures display changes in both measures with 95% confidence intervals.
- Layer coverage: Figures 18–22 cover True-False neurofeedback at the 50th, 0th, 25th, 75th, and 100th-percentile layers.
- Effect sizes: Figure 8 summarizes final-turn effect sizes across normalized layer depth using Cohen’s d for probe output and Cohen’s h for self-report proportion.
- Outcome measures: For True-False, the left plots show changes in mean probe output measuring truthfulness.
- Outcome measures: The right plots show changes in the proportion of cases where the model self-reported label 1, interpreted as true.
- Uncertainty: Shaded regions in the True-False plots denote 95% confidence intervals.