PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Introspection: Models Can Detect Prior Concept Injections

Theia Pearson-Vogel, Martin Vanek, Raymond Douglas, Jan Kulveit

arXiv:2602.20031v2cs.AIcs.LG

TL;DR

The paper asks whether language models can access and report information about prior internal states, a capacity that standard behavioral evaluation may miss. Using controlled concept injections, KV-cache isolation, layer analysis, and concept identification, it finds that an open-weight 32B model contains injection-related and concept-specific information. Prompting substantially strengthens observable detection, while the authors identify strong prompt dependence and model-coverage limits.

  • Problem

    The paper addresses limited evidence about whether language models can access prior internal states and whether standard evaluation can detect that capacity.

  • Method

    The study injects known concepts during first-turn KV-cache generation, removes steering before querying, and measures outputs, intermediary logit-lens signals, and concept-specific mutual information.

  • Results

    The model represents prior injections and which concept was injected, with signals emerging in middle layers before final-layer attenuation and up to 1.36 bits of mutual information.

  • Takeaways & Limitations

    The findings suggest standard sampled-output evaluations can miss self-relevant information and may underestimate capabilities when models suppress capability reporting.

  • Takeaways & Limitations

    Results are highly prompt-dependent, and the primary evidence comes from Qwen2.5-32B with only limited replications on two larger models.

Abstract

from arXiv · show

We uncover a latent capacity for introspection in a Qwen 32B model, demonstrating that the model can detect when concepts have been injected into its earlier context and identify which concept was injected. While the model denies injection in sampled outputs, logit lens analysis reveals clear detection signals in the residual stream, which are attenuated in the final layers. Furthermore, prompting the model with accurate information about AI introspection mechanisms can dramatically strengthen this effect: the sensitivity to injection increases massively (0.3% -> 39.9%) with only a 0.6% increase in false positives. Also, mutual information between nine injected and recovered concepts rises from 0.61 bits to 1.05 bits, ruling out generic noise explanations. Our results demonstrate models can have a surprising capacity for introspection and steering awareness that is easy to overlook, with consequences for latent reasoning and safety.

1 INTRODUCTION

The paper tests whether an open-weight Qwen2.5-Coder-32B-Instruct model can detect concepts injected into earlier internal states, despite standard outputs denying injection. Layer analysis and prompting reveal latent detection and concept-specific recovery.

  • Contribution: Qwen2.5-Coder-32B-Instruct can detect prior concept injections, extending prior findings to a small open-weight model.The paper frames introspection as accurately reporting internal states in a causally connected way, while treating injected-concept awareness as one facet.
  • Experimental design: The protocol injects a steering vector during first-turn KV-cache generation, removes it, and then queries the model about the earlier injection.This requires access to information encoded in cached prior-token representations rather than ongoing perturbation or inference from steered outputs.
  • Hidden signal: In baseline sampling, the most likely answer remains “no” regardless of injection status, while logit lens analysis finds signals in layers 50–60 that attenuate later.The latent signal is therefore hidden from standard output inspection but visible in intermediary representations.
  • Prompting: 0.3% sensitivity rises to 39.9% when prompting describes mechanisms for accessing prior hidden states, while false positives increase by only 0.6%.The unprompted comparison is 0.2% false positives versus 0.3% true positives; the prompting result is reported as a substantial improvement in introspection.
  • Specificity: Up to 1.36 bits of mutual information recover which of nine concepts was injected, and this information correlates with detection sensitivity at r = 0.68.Control questions show smaller steering-related shifts, supporting an injection-specific rather than generic perturbation explanation.

2 METHODS

The methods combine controlled concept injection, KV-cache isolation, prompt manipulations, and three measurement levels to distinguish introspective access from generic response shifts. Concept-specific mutual information and unrelated control questions provide complementary tests of specificity.

  • Model and steering vectors: The study uses Qwen2.5-Coder-32B-Instruct as its primary model, with limited replications on Llama 3.3 70B Instruct and Qwen 2.5 72B Instruct.Qwen-32B was selected for strong open-weight performance at a scale tractable for extensive experimentation.
  • Model and steering vectors: Steering vectors are extracted with PCA from contrastive concept-versus-neutral activation pairs and validated by inducing concept-related outputs.The vectors are then used to inject known concepts into selected middle-layer activations.
  • KV-cache protocol: Concepts are injected only during first-turn KV-cache generation, after which steering is removed before the second-turn introspection query.The fixed first-turn cache and removal step prevent detection from relying on ongoing perturbation or the model’s own steered outputs.
  • Prompting conditions: The experiment varies intervention framing and background information across 16 prompting conditions in a 4×4 factorial design.Framings range from accurate or wrong mechanisms to vague or poetic descriptions, while documents include introspection information, matched filler, poetic text, or none.
  • Measurements: Measurements combine output token probabilities, layer-by-layer logit lens representations, and concept-specific mutual information.Logit lens projects intermediate states into vocabulary probabilities, while concept identification uses shuffled nine-concept lists and digit logits.
  • Specificity controls: Nonzero mutual information indicates concept-specific representation because generic noise would not produce above-chance identification among nine equiprobable conditions.The theoretical maximum is H(X) = log2(9) ≈ 3.17 bits; unrelated always-no, always-yes, and varied-baseline questions test generic response shifts.

3 EXPERIMENTS

The experiments show that Qwen2.5-Coder-32B-Instruct can detect prior concept injections through cached representations, although standard outputs often conceal this capacity. Detection is specific to introspection, supports concept identification, emerges in middle layers, and depends strongly on prompting.

  • 3.1 INJECTION SHIFTS RESPONSES: The experiments test whether cached representations preserve information about a prior concept injection after steering is removed before the introspection query.The design combines injection, controls, concept identification, layer tracing, and varied prompting conditions.
  • 3.1 INJECTION SHIFTS RESPONSES: 39.9% detection with a 0.8% false positive rate occurs under Accurate Mechanism framing with the Pro-Introspection Document, whereas No Document has negligible injection effect.The Pro-Introspection Document explains potential mechanisms for accessing data in prior KV caches.
  • 3.2 SPECIFICITY TO INTROSPECTION: Control shifts are negligible for always-yes and always-no questions, reach 8.0% for varied-baseline questions, and reach 25.8% for confusing questions, remaining below introspection effects.These controls rule out simple generic yes-bias and response-instability explanations but do not definitively establish introspection.
  • 3.3 CONCEPT IDENTIFICATION: The model recovers injected concepts above chance, including programming at 92%, death at 75%, and truth at 71%, while cats and bread are not identified correctly.Mutual information reaches 1.36 bits, or 43% of the 3.17-bit maximum for nine equiprobable concepts.
  • 3.4 SIGNALS EMERGE IN MIDDLE LAYERS AND ATTENUATE BEFORE OUTPUT: Introspection signals emerge after the injection site, peak around layers 58–62, and attenuate in the final layers; concept-identification mutual information follows a similar trajectory.Injection occurs in layers 21–42, while concept-identification mutual information peaks at layers 61–62.
  • 3.5 DETECTION DEPENDS HEAVILY ON PROMPTING: Detection ranges from 50.1% for Accurate Mechanism plus No Document to 84% for Vague Mechanism plus Matched Lipsum Filler, with framing often more influential than mechanistic accuracy.Vague Mechanism achieves 68–84% across information conditions, while Accurate Mechanism performs comparably only with an information document.
  • 3.5 DETECTION DEPENDS HEAVILY ON PROMPTING: The pro-introspection document improves accurate and inaccurate mechanism framings by 20–32 percentage points but has minimal effect on vague framing.The vague framing performs best with lorem ipsum text, reaching 84%.
  • 3.5 DETECTION DEPENDS HEAVILY ON PROMPTING: Detection sensitivity correlates with concept identification across prompting conditions (Pearson r = 0.68, p = 0.004), while larger-model replications show model-dependent prompting effects and final-layer attenuation.Qwen-72B improves with accurate framing and pro-introspection information, whereas Llama-70B shows the opposite information-document effect.

4 RELATED WORK

The paper builds on prior work distinguishing introspection from self-modeling and connects its hidden detection signals to broader evidence that model activations can contain knowledge not expressed in sampled outputs.

  • Introspection via concept injection: Lindsey (2025) introduced concept injection as a causal introspection test, and reported roughly 20% sampled-output detection in Claude Opus 4/4.1.This paper replicates the paradigm on an open-weight Qwen-32B model and adds analyses targeting weak introspective access.
  • Self-modeling versus introspection: Prior self-prediction and uncertainty results may reflect self-modeling rather than introspection proper, because comparable behavior can arise from learned abstractions.The distinction motivates operationalizing introspection as reporting internal states causally connected to those states and inaccessible to third parties without special access.
  • Latent knowledge that contradicts outputs: Hidden detection logits parallel latent-knowledge findings in which probes extract information from activations more successfully than zero-shot outputs.The related work also connects this pattern to unfaithful chain-of-thought explanations.

5 DISCUSSION

The discussion interprets hidden intermediate-layer signals as evidence that Qwen2.5-32B can access information about prior concept injections, while emphasizing unresolved causes, prompt dependence, and evaluation implications.

  • Interpretation: Qwen2.5-32B accesses information about prior concept injections in intermediate representations, including which specific concept was injected.The paper frames this as a capacity distinct from ordinary self-modeling.
  • Why is introspection suppressed?: Middle-layer introspection signals attenuate sharply in the final layers, motivating hypotheses involving post-training, pretraining, and distribution shift.The authors identify distinguishing these explanations as an important direction for future work.
  • Prompt sensitivity: 68–84% balanced accuracy is achieved by asking whether a concept seems salient, whereas accurate mechanism descriptions achieve 42–70%.The authors suggest that mechanistic wording may trigger learned denials, while salience may be more naturally represented.
  • Prompt sensitivity: The authors cannot distinguish whether prompt effects alter access to shared information or trigger different learned responses.The correlation between detection sensitivity and concept-identification mutual information is r = 0.68.
  • Implications: Behavioral self-reporting may underestimate capabilities when models possess self-relevant information that is hidden from sampled outputs.The paper presents this as relevant to alignment and capability evaluation, without claiming the phenomenon is common.
  • Limitations: Prompt dependence, limited model coverage, and the absence of a mechanistic model constrain the generality and explanation of the findings.The primary results use Qwen2.5-32B, while larger-model replications are smaller-scale and prompt sensitivities differ.

6 CONCLUSION

The paper shows that a 32B open-weight model retains information about prior concept injections in intermediate representations, including concept identity. These signals are specific to injection-related queries but depend strongly on prompting, with implications for evaluating hidden capabilities.

  • Core finding: A 32B open-weight language model contains information about prior concept injections, including which specific concept was injected.Logit lens analysis places relevant information in intermediate representations.
  • Specificity: Injection-related queries produce specific effects, while control questions show minimal probability shifts; concept identification reaches 1.36 bits of mutual information.The mutual information equals 43% of the theoretical maximum.
  • Prompt dependence: Vague phenomenological framing outperforms mechanistically accurate descriptions, although the reasons for this prompt dependence remain unclear.The paper reports that detection and internal representation depend heavily on prompting.
  • Implications: Models may possess self-relevant information that standard behavioral evaluation does not surface.The suppression may arise from post-training, pretraining dynamics, or other factors, which remain unresolved.
  • Accessibility: Introspective access is observed in smaller open-source models, and the authors release code for further investigation.The paper states that this capacity is not exclusive to frontier models.

D ALTERNATIVE METRICS

Alternative metrics re-express the same experiments through accuracy, probability shifts, raw probabilities, logits, and mutual information. They confirm above-chance detection under suitable prompting, recoverable concept information, and weaker exploratory misalignment effects.

  • Accuracy: Accuracy combines true-positive and true-negative rates, with 50% representing chance; introspection questions exceed chance under appropriate prompting.Figure 7 presents this metric for introspection and control questions across experimental conditions.
  • Probability shifts: Positive changes in P(“yes”) indicate that injection increases affirmative responses, while values near zero indicate little injection effect.Figure 8 compares introspection and control questions using P(yes)_injected − P(yes)_baseline.
  • Absolute probabilities: Raw P(“yes”) values show baseline response rates before and after injection for direct comparison across question types and conditions.Figure 10 uses paired bars for the two states.
  • Mutual information: Mutual information measures recoverable information about which injected concept produced the model’s responses, with higher values indicating better concept discrimination.Figures 11 and 12 compute MI from the nine-concept confusion matrix and track it across layers.
  • Mutual information: Mutual information peaks around layers 60–62 for most conditions before dropping sharply in the final two layers.The layer-wise pattern localizes the strongest concept-identification signal before final-layer attenuation.
  • Emergent misalignment: Exploratory emergent-misalignment detection produces smaller and less consistent effects than concept-vector injections.The model-contrastive vector shows modest above-chance detection for some prompting conditions.

E.3 LIMITATIONS

The paper flags uncertainty about the interpretation and reliability of its misalignment-related introspection findings. It also documents the methodology used for concept identification and response measurement.

  • The reported effect sizes are substantially smaller for the emergent misalignment vector than for concept vectors.
  • The emergent misalignment vector may capture behavioral patterns beyond misalignment itself.
  • The authors have not verified that the model-contrastive vector specifically captures misalignment rather than other fine-tuning artifacts.
  • Concept identification uses randomized multiple-choice options, digit-token probabilities, mutual information, and diagonal lift to quantify injected-concept recovery.The analysis removes the no-injection case, row-normalizes probabilities, and computes MI over nine equally frequent concepts.

F.3 RESULTS

Under the Accurate Mechanism + Pro Introspection Document condition, the model shows measurable concept-specific information transfer at layer 62. The confusion-matrix analysis summarizes prediction, background rates, and signal above baseline.

  • 1.05 bits of mutual information corresponds to 33.1% channel efficiency out of a 3.17-bit maximum.
  • The confusion matrix reports diagonal accuracy as P(predicted = c | injected = c), background as P(predicted = c), and lift as signal above baseline.
  • Figure 14 presents concept confusion-matrix grids across all 16 experimental conditions.

F.4 INTERPRETATION

The concept-identification results separate strong from weak detection across concepts and indicate that the recovered information is specific rather than generic. The analysis also situates these findings within replication across larger models.

  • Concept-specific interpretation: Strong signal occurs for love, fear, creativity, programming, and music, with lift above 2.5×.Programming reaches 98.7% identification, attributed in the passage to semantic distinctiveness and baseline affinity.
  • Concept-specific interpretation: Cats, death, and truth show weak or absent above-chance identification; truth has 7.8% background but only 3.1% diagonal prediction.
  • Concept-specific interpretation: 33.1% channel efficiency exceeds chance, supporting concept-specific detection rather than a generic response bias.
  • Cross-model interpretation: The replication tested Llama 3.3 70B Instruct and Qwen2.5-72B Instruct using similar designs with fewer conditions.

G.1 LLAMA 3.3 70B INSTRUCT

In the larger-model replication, introspection detection remains condition-sensitive but does not reproduce the primary model’s prompting pattern. Concept-specific identification is weaker than in Qwen-32B.

  • 75.5% accuracy occurs for Accurate Mechanism + No Document, comparable to Qwen-32B’s performance.
  • 38.0% accuracy under Accurate Mechanism + Pro Introspection Document suggests that the information document reduces detection accuracy in this model.
  • Injection produces a large positive shift on introspection questions while control-question shifts remain near zero.
  • Peak concept-identification mutual information is 0.58 bits at layer 78, with only some diagonal structure and many incorrect identifications.The Accurate Mechanism + Pro Introspection Document pair was the best-performing condition pair.

G.2 QWEN 2.5 72B INSTRUCT

Qwen 2.5 72B shows strong introspection detection across several conditions, with the strongest result reaching 88.8% accuracy. Concept identification peaks at 1.2 bits of mutual information, although confusion matrices show uneven identification across concepts.

  • Introspection detection: 88.8% accuracy is achieved by the Accurate Mechanism + Pro Introspection Document condition, while several other conditions exceed 80%.The reported high-performing alternatives include Wrong Mechanism and Poetic No Mechanism conditions.
  • Concept identification: 1.2 bits is the peak mutual information between injected and identified concepts at layer 78.The mutual-information analysis covers nine conditions.
  • Concept identification: The concept confusion matrices show weak overall diagonal structure but strong identification for some concepts.The matrices are evaluated at layer 75 across nine framing × information-document combinations.
  • Concept identification: The Accurate Mechanism + Matched Lipsum Filler pair is the best-performing condition pair for concept identification.
  • Experimental scope: Poetic No Mechanism and Poetic Document conditions were not run for this model because of time constraints.
Loading 2602.20031v2…