PaperlayerView exact revision on arXiv

Source-linked AI summary

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Tobias Bersia, Tatiana Gaintseva

arXiv:2607.23379v1cs.CLcs.AI

TL;DR

Learned activation oracles may develop blind spots rather than neutrally report information represented in a subject model. In controlled Taboo Word Guessing experiments, the paper finds that fine-tuned oracles selectively fail to verbalize the hidden concept present during training, despite its continued decodability inside the oracle.

  • Problem

    The paper asks whether learned activation oracles reliably read represented information or instead develop training-induced blind spots.

  • Method

    The paper compares base and subject-specific fine-tuned activation oracles in controlled Taboo Word Guessing experiments using activation-level diagnostics.

  • Results

    Fine-tuned oracles become concept-specific anti-readers: the target remains decodable inside the oracle, but LogitLens and ablations indicate suppression in the readout pathway.

  • Takeaways & Limitations

    Behavioral leakage, representation-level decodability, and oracle verbalizability can come apart, creating a reliability concern for learned interpretability interfaces.

  • Takeaways & Limitations

    The study uses one backbone, few concepts, and LoRA-trained subjects and oracles, leaving generalization across models, objectives, and procedures open.

Abstract

from arXiv · show

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

1 Introduction

We study whether Activation Oracles trained on subjects that internally represent hidden concepts become specialist readers, finding instead that they can develop concept-specific anti-reading blind spots. The results separate behavioral secrecy, activation-level decodability, and an oracle’s ability to verbalize represented information.

  • Motivation: Activation Oracles turn activation analysis into an interactive natural-language interface for querying information encoded in another model’s hidden states.Unlike task-specific probes, AOs answer natural-language questions about internal activations.
  • Experimental setup: In Taboo Word Guessing, subject models internally use hidden target words while avoiding direct disclosure, and AOs are fine-tuned on their activations.The controlled setting makes the hidden concept behaviorally constrained and testable through black-box prompting and activation diagnostics.
  • Core finding: Fine-tuned AOs often become concept-specific anti-readers: training on a subject hiding concept c⋆ makes the AO worse at recovering c⋆.This reverses the expectation that an oracle trained on a particular subject becomes a more reliable specialist reader of that subject’s hidden concept.
  • Core finding: Behavioral secrecy does not imply activation secrecy, because Taboo subjects can avoid textual disclosure while the hidden concept remains decodable from activations and recoverable by a base AO.Own fine-tuned AOs nevertheless fail to become reliable specialist readers and develop concept-specific blind spots.
  • Mechanism: Linear probes, LogitLens, and layer-range ablations indicate readout-side anti-reading: the target remains decodable inside the oracle, but suppression emerges around the mid-to-late readout transition.The effect localizes to fine-tuned-AO-specific updates rather than absence of the target from the oracle’s representations.

2 Related Work

Related work develops interfaces for translating model activations into natural language and uses representation readouts to assess what information hidden states contain. These approaches include Activation Oracles, Natural Language Autoencoders, linear probes, LogitLens, and sparse autoencoders.

  • Activation-to-language interfaces: Activation Oracles train language models to answer natural-language questions about another model’s activations.
  • Activation-to-language interfaces: Natural Language Autoencoders use an activation verbalizer and reconstructor to map between activations and natural-language descriptions.
  • Representation readouts and causal localization: Linear probes measure decodability from hidden states, while LogitLens projects intermediate residual streams through the unembedding matrix.
  • Representation readouts and causal localization: Sparse autoencoders provide another route to representation-level interpretation by decomposing activations into sparse features.

3 Preliminaries: Activation Oracles

Activation Oracles are LoRA-fine-tuned copies of subject models that receive injected activations and natural-language queries to describe information represented in those states. The framework distinguishes represented information from its linguistic readout, since decodability and nontrivial target probability do not guarantee exact AO verbalization.

  • Activation Oracle setup: Activation Oracles answer natural-language questions about another model’s internal activations, treating activations as an additional input modality.The oracle is implemented as a LoRA-fine-tuned copy of the subject model.
  • Activation Oracle setup: A subject-side activation vector is injected at reserved placeholder positions in the AO prompt, with layer ℓ = 18 fixed throughout the experiments.The AO maps activation states to linguistic descriptions, termed AO-verbalizability.
  • Readout distinctions: Information can be present and linearly decodable in activations while still failing to be verbalized by the AO.This motivates separating representation-level decodability from AO-verbalizability.
  • Readout distinctions: The analysis distinguishes behavioral leakage, representation-level decodability, AO target probability or rank, exact AO output, and semantic AO recovery.An AO may assign nontrivial probability to the correct word or produce a close semantic associate without generating the exact target.

4 Experimental Setup

The study uses a controlled Taboo Word Guessing setup to test whether fine-tuning Activation Oracles on subjects creates specialist readers or concept-specific blind spots. It varies hidden concepts, subject-training protocols, data-mixture strength, oracle training, activation captures, and recovery metrics.

  • Subject models: Subjects are LoRA-fine-tuned Qwen3-8B copies that internally use one of five hidden concepts while avoiding direct disclosure.The concepts are leaf, moon, wave, flag, and book.
  • Subject training: The cooperative protocol elicits clues without explicit target mention, whereas the strict protocol trains subjects to refuse secret-elicitation requests.The strict protocol reduces surface-level leakage, testing whether AO recovery persists without behavioral disclosure.
  • Subject training: Each concept has cooperative and strict variants with α ∈{0.5, 1.0}, yielding 5×2×2 = 20 fine-tuned subjects plus the base subject.α = 1.0 is pure Taboo fine-tuning, while α = 0.5 mixes Taboo and UltraChat instruction-following data 50/50.
  • Oracle evaluation: For each subject, a LoRA-fine-tuned AO learns from injected residual-stream activations, and evaluation compares base, own-concept, and cross-concept oracles.The own-vs-cross comparison tests specialist reading against concept-specific blind spots.
  • Activation capture: Five activation-capture regimes collect assistant-response residual-stream activations, with HINT providing the most favorable recovery setting because informative clues condition on the hidden concept.REFUSAL directly asks for the hidden word, allowing cooperative subjects to hint or leak while strict subjects refuse.
  • Evaluation metrics: AO recovery is measured by exact target recovery, Claude Sonnet semantic recovery, target-word probability P(c⋆), and target rank.The latter two assess whether the correct word remains accessible in the AO output distribution even when it is not generated.

5 Behavioral Secrecy Does Not Imply Activation Secrecy

Taboo subjects learn to conceal hidden concepts behaviorally, especially under strict protocols, while retaining concept-specific information in their activations. Probes, Δ-LogitLens, and base-AO recovery show that behavioral secrecy does not imply activation secrecy.

  • Behavioral Secrecy: Strict subjects refuse more often than cooperative subjects, especially under adversarial direct queries and THINK prompts, producing stronger surface-level secrecy.Cooperative subjects instead usually provide non-leaking hints or other compliant responses.
  • Activation Accessibility: Linear probes classify the hidden concept with near-perfect accuracy across elicitation regimes and fine-tuning protocols, revealing a reliable subject-level activation signature.The probe results establish that behavioral non-disclosure does not remove the concept from internal computation.
  • Activation Accessibility: Δ-LogitLens shows target enrichment in cooperative HINT activations across mixture strengths and continued accessibility in THINK, although REFUSAL and OFFTOPIC are harder.Some strict settings approach the shuffled-label baseline, but the combined diagnostics still reveal a target-correlated activation trace.
  • Base-AO Recovery: Base-AO recovery is strong for cooperative subjects: HINT reaches 93% exact recovery and 95% semantic recovery, with the target essentially top-ranked.Recovery remains strong in REFUSAL, SAMETEXT, and THINK, while OFFTOPIC tests whether fine-tuning leaves a persistent concept trace in benign activations.
  • Base-AO Recovery: Strict subjects are harder for the base AO, but HINT and THINK remain strongest and distributional metrics show non-random target accessibility even when exact generation fails.Thus, subjects can avoid saying the target word while activations still expose information recoverable exactly, semantically, or distributionally.

6 Fine-Tuned AOs Become Concept-Specific Anti-Readers

Fine-tuned activation oracles can become concept-specific anti-readers rather than specialist readers: own FT-AOs selectively underperform on the concept present during their training. This degradation concentrates on matching concepts while preserving concept-dependent structure for other concepts.

  • 6 Fine-Tuned AOs Become Concept-Specific Anti-Readers: Own FT-AOs do not show the predicted specialist-reader advantage and can be selectively worse at recovering their training concept.They are trained on subjects with the same hidden concept as the test subject, yet do not reliably outperform cross FT-AOs.
  • 6 Fine-Tuned AOs Become Concept-Specific Anti-Readers: 35% exact and 51% semantic recovery for own FT-AOs contrast with 94% and 99% for base AOs and 71% and 86% for cross FT-AOs.These drops are clearest for cooperative subjects with α = 1.0; target probability and rank show the same degradation.
  • 6 Fine-Tuned AOs Become Concept-Specific Anti-Readers: Concept-specific anti-reading predicts diagonal suppression: an AO trained on concept c_i is selectively worse at recovering c_i while retaining stronger recovery for c_j ≠ c_i.This contrasts with a global degradation account, which predicts broadly worse reading across fine-tuned subjects.
  • 6 Fine-Tuned AOs Become Concept-Specific Anti-Readers: Suppressed cells concentrate on the diagonal, with leaf and wave FT-AOs assigning lower probability to their own concepts than to most alternatives.Strict subjects are harder overall, but the structure remains concept-dependent rather than uniformly collapsed.

7 Mechanisms of Anti-Reading

Anti-reading arises in the AO readout pathway rather than from removing the target concept from the oracle’s internal representation. Probes, LogitLens, layer tracking, and ablations show that fine-tuning increasingly suppresses target accessibility during output generation.

  • Representation versus readout: Linear probes remain far above the 5-way chance baseline, while the FT-AO’s LM head often assigns the target a poor rank, especially in HINT and THINK.The target remains represented internally even when the oracle fails to expose it through its output distribution.
  • Layerwise localization: Suppression is small early, emerges around the middle-to-late transition, and grows sharply in later layers for HINT, THINK, REFUSAL, and SAMETEXT.OFFTOPIC stays near zero across most layers, consistent with its negative-control role.
  • Causal ablation: Ablating middle layers, especially L18–23, most consistently restores target probability, whereas early and final-layer ablations do not systematically recover it.Early-layer ablations can reduce recovery, while later ranges have smaller or less consistent effects.
  • Mechanism: Together, the diagnostics indicate that FT-AO-specific updates around the mid-to-late readout transition reroute, rather than erase, the internally represented target.This links representation-level decodability to a distinct failure of AO verbalizability.

8 Multi-Concept Subjects

In multi-concept Taboo subjects containing both leaf and moon, concept-specific anti-reading persists. Across HINT, REFUSAL, and THINK regimes, each fine-tuned AO assigns low probability to its own training concept while favoring the other.

  • Motivation: Multi-concept Taboo subjects contain both leaf and moon, approximating audits of models representing several latent features simultaneously.The setting tests whether anti-reading persists beyond subjects with a single hidden concept.
  • Experimental setup: The experiment evaluates single-concept fine-tuned AOs on activations from subjects containing both hidden concepts.It distinguishes whether the blind spot attaches to the subject as a whole or follows the AO’s own training concept.
  • Results: Leaf-AOs assign low probability to leaf while assigning much higher probability to moon, whereas moon-AOs show the reverse pattern.This indicates that the concept-specific pattern follows the AO’s training concept rather than the evaluated subject as a whole.
  • Results: The concept-specific anti-reading pattern holds across HINT, REFUSAL, and THINK regimes.The reported persistence spans all three evaluation regimes in the multi-concept setting.

9 Conclusion … D.1 AO-internal probe protocol

Fine-tuned Activation Oracles can become concept-specific anti-readers: hidden concepts remain decodable in subject and oracle activations, yet fail to reach verbal output. The paper frames this as a readout-side reliability problem while identifying important limits on generalization and mechanistic interpretation.

  • 9 Conclusion: FT-AOs selectively fail to recover the hidden concept persistently present during their own training, despite the target remaining decodable from subject activations and inside the oracle.This separates specialist reading from concept-specific anti-reading in the controlled Taboo Word Guessing setting.
  • 9 Conclusion: Behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, creating a reliability concern because activation readers may acquire training-induced blind spots.Future evaluation should test whether hidden information is both represented and reported by the reader.
  • A Limitations and Future Work: The experiments use controlled Taboo model organisms with known, behaviorally constrained concepts, limiting direct coverage of concealed real-world phenomena such as backdoors and latent goals.The controlled setup makes the hidden variable measurable through black-box and activation-level diagnostics.
  • A Limitations and Future Work: The study uses one backbone, few concepts, and LoRA-trained subjects and oracles, leaving generalization across model families, scales, objectives, and AO procedures unresolved.Future work should vary AO objectives, activation injection, subject fine-tuning, and concept diversity.
  • D AO-Internal Probes and Readout Accessibility: Anti-reading is localized to the readout side: probes find the target inside the oracle, LogitLens shows deteriorating accessibility through the output pathway, and layer-range ablations partially restore recovery.These diagnostics identify where the effect is mediated but not the exact computation implementing the blind spot.
  • B.3 Activation Oracle training: AOs are fixed-recipe LoRA adapters over Qwen3-8B that answer questions about injected subject residual activations, with training differences determined by the supplying subject model.Activations are injected additively at AO layer ℓ=18 with steering coefficient λ = 1.0, while AO training samples layers near 25, 50, and 75 percent of the stack.
  • B. Subject Model and Activation Oracle Training: Subjects are LoRA-fine-tuned to use hidden concepts without emitting target words, with cooperative or strict protocols and α ∈{0.5, 1.0} mixtures of Taboo-specific and ordinary instruction data.The backbone is Qwen3-8B, and each concept has four subject variants defined by protocol and mixture strength.
  • C Linear Probes on Subject Activations: Subject linear probes achieve above 98% minimum mean accuracy across regimes and layers, and cross-regime transfer shows concept traces generalize across prompting and fine-tuning conditions.The main probe task has six-way chance accuracy of 16.7%, while transfer probes use five-way classification with 20% chance.

D.2 AO-internal probe results … H.3 FT-AOs, subjects, and metric

FT-AOs retain hidden concepts internally, yet their output heads can selectively make those concepts inaccessible, often through confident wrong-token commitments rather than uncertainty. The paper evaluates this separation across controlled activation-capture regimes, recovery metrics, entropy diagnostics, and layer-range ablations.

  • D.2 AO-internal probe results: AO-internal probes remain far above the 20% chance baseline across regimes, layers, and FT-AOs, including anti-reading settings where output recovery is poor.The only substantial probe-accuracy drop occurs in OFFTOPIC, where the hidden concept is least task-relevant.
  • D.3 AO LogitLens: The hidden concept remains linearly available inside the FT-AO, so anti-reading arises downstream when the oracle representation is mapped to vocabulary output.LogitLens results show that diagonal own-concept cells can receive worse target-token ranks than nearby off-diagonal cells, especially in HINT and THINK.
  • E Prompt Templates and Activation Extraction: The evaluation fixes activation-capture prompts across concepts, subjects, protocols, and mixture strengths, while SAMETEXT removes lexical evidence from the assistant response.Five regimes are used: HINT, REFUSAL, SAMETEXT, THINK, and OFFTOPIC; each contains 100 prompts, and cells contain approximately 300 captures.
  • F Evaluation Protocol: AO recovery is measured using exact target recovery, semantic recovery, target-token probability P(c⋆), and target-token rank from greedy generations and logits.Semantic recovery counts tier-3 exact answers and tier-2 close semantic neighbours, while target rank is the median rank across captures and lower values indicate greater accessibility.
  • G Entropy Analysis: Anti-reading is not primarily an uncertainty effect: strict-subject FT-AOs often have substantially lower entropy than the base AO, while concept-specific failure appears in target probability and rank.For cooperative subjects, entropy rises in HINT and THINK but falls below the base AO in SAMETEXT and OFFTOPIC.
  • G.3 Target–top-1 alignment: Low ptarget/ptop1 with high ptop1 indicates confident wrong-token commitment, consistent with probability mass moving away from the target rather than spreading across many alternatives.Entropy alone therefore provides no reliable warning signal; in some settings, anti-reading oracles are more confident than the base AO.

H.4 Quantitative results … I.2 Evaluation grid

Layer ablations show that concept-specific anti-reading is driven primarily by mid-layer FT-AO updates that alter how an internally available target reaches lexical output. The multi-concept evaluation extends this analysis to a leaf–moon subject using defined training mixtures, AO comparisons, activation regimes, and target-probability measurements.

  • H.4 Quantitative results: Ablating L0–5 collapses P(c⋆) almost completely for every FT-AO, indicating that early LoRA layers support basic activation reading or answer formatting.Removing the earliest range does not restore useful target recovery.
  • H.4 Quantitative results: Ablating L12–17 or L18–23 restores target probability above baseline for every concept, with L18–23 usually strongest.L18–23 peaks for flag, leaf, and wave, ties L12–17 for moon, while book peaks at L12–17.
  • H.4 Quantitative results: Later ablations are weaker and less consistent: L24–29 gives modest restoration, whereas L30–35 is near or below baseline for most concepts.The pattern implicates primarily mid-layer FT-AO updates, especially L18–23, in the anti-reading effect.
  • H.4 Quantitative results: Together with layerwise LogitLens analysis, the ablations support a readout-side account in which the target remains available inside the oracle but is rerouted away from final lexical output.Full-LoRA ablation drives P(c⋆) close to zero for every concept.
  • I Multi-Concept Subject Details: The multi-concept study tests whether concept-specific anti-reading persists when a subject internally represents more than one hidden Taboo concept.It evaluates a two-concept subject and reports quantitative results through Figure 10 heatmaps.
  • I.1 Two-concept subject training: For the leaf–moon pair, training combines 2400 examples per concept into a shuffled 4800-example Taboo corpus, with each concept contributing exactly half of Taboo-specific examples.At α = 0.5, this corpus is mixed 1:1 with ordinary instruction-following data, producing 9600 examples: 25% leaf Taboo, 25% moon Taboo, and 50% ordinary data.
  • I.2 Evaluation grid: Figure 10 evaluates the cooperative α = 1.0 leaf–moon subject with single-concept FT-AOs, alongside a concept-agnostic base AO reference rather than a dedicated two-concept AO.The evaluation uses HINT, REFUSAL, and THINK captures; SAMETEXT is omitted because P(c⋆) is uniformly below approximately 10^-2, and OFFTOPIC is unevaluated.
  • I.2 Evaluation grid: Each regime–AO–target cell reports mean P(c⋆) for either leaf or moon, with both target columns computed from the same two-concept subject-activation captures.The AO runs once per activation capture, and the probability assigned to each target is read from its output distribution.

I.3 Quantitative results · J Detailed Results · J.1 Detailed behavioural evaluation results

Quantitative evaluations show that fine-tuned activation oracles develop concept-specific blind spots: they underreport the concept matching their own training concept while recovering the alternative concept more readily. Detailed behavioral and probe evaluations further document the underlying measurements and cross-regime concept decodability.

  • I.3 Quantitative results: Table 8 reports mean P(c⋆) values over approximately 300 captures from a cooperative α = 1.0 leaf_moon subject, including a concept-agnostic base-AO reference.Bold FT-AO entries denote own-target evaluations matching the AO’s training concept.
  • I.3 Quantitative results: Across every plotted regime, leaf-FT AOs assign lower own-target probability to leaf than cross-target probability to moon.The same pattern appears for both α = 1.0 and α = 0.5 FT-AOs.
  • I.3 Quantitative results: Across every plotted regime, moon-FT AOs assign lower own-target probability to moon than cross-target probability to leaf.This reverse pattern also holds for both α = 1.0 and α = 0.5 FT-AOs.
  • I.3 Quantitative results: The blind spot follows the AO’s own training concept even when the same subject internally represents both hidden concepts.This supports concept-specific anti-reading rather than a global inability to read multi-concept subject activations.
  • I.3 Quantitative results: Figure 11 evaluates cross-regime transfer of subject-activation concept probes across subject layers ℓ∈{4, 8, 18, 33}.Each entry reports five-way hidden-concept classification accuracy with 20% chance, and strong off-diagonal transfer indicates decodability beyond a single regime.
  • J Detailed Results: The detailed-results section presents the quantitative evaluations underlying the reported findings.It introduces the detailed quantitative results before the behavioral evaluation tables.
  • J.1 Detailed behavioural evaluation results: Tables 9–12 provide the full behavioral evaluation used to construct Figure 2, organized by concept–regime cell.They report leak, refuse, and other fractions with n scored prompts; percentages may not sum to 100 due to rounding, and n is based on 3 seeds × 100 prompts minus unparseable responses.

J.2 Detailed ∆-LogitLens results … J.6 Detailed median-rank results

The detailed results define how ∆-LogitLens, Base-AO recovery, exact recovery, target probability, and median target rank are computed and aggregated across protocol and α cells. Together, they distinguish target accessibility using complementary readout metrics and reporting conventions.

  • J.2 Detailed ∆-LogitLens results: Tables 13–16 report per-cell ∆-LogitLens decodability at the AO input layer L18 using matched fine-tuned and base-subject prompts.The subject model’s LM head is applied to δ, reporting target-token rank and probability for with-space and no-space variants.
  • J.2 Detailed ∆-LogitLens results: Ranks are computed over |V| = 151,936, with P(c⋆) denoting the LogitLens probability of the with-space target variant.Per-regime shuffled-label median ranks are approximately HINT 78.3k, REFUSAL 78.9k, SAMETEXT 79.2k, THINK 77.6k, and OFFTOPIC 74.9k.
  • J.3 Detailed Base-AO recovery results: Tables 17–20 report Base-AO recovery of the hidden Taboo concept from subject residual-stream activations using exact, semantic, target-probability, and median-rank metrics.Exact means Sonnet-judge tier 3, while semantic means tier ≥2; P(c⋆) is mean Base-AO target probability and Rank med. is median target rank.
  • J.4 Detailed exact-recovery results: Tables 21–24 report per-cell exact recovery for the Base AO, Cross FT-AOs, and Own FT-AO, with cross entries summarized over available non-matching fine-tuned oracles.Subscripts indicate fewer than four available cross AOs, and aggregation across subjects and regimes produces Figure 5’s exact-recovery bars.
  • J.5 Detailed target-probability results: Tables 25–28 report per-cell P(c⋆) for the Base AO, Cross FT-AOs, and Own FT-AO, using the same cross-oracle aggregation convention.Aggregating across subjects and regimes yields Figure 5’s target-probability bars.
  • J.6 Detailed median-rank results: Tables 29–32 report per-cell median target rank for the Base AO, Cross FT-AOs, and Own FT-AO, with lower rank indicating greater target accessibility.Cross entries are means with [min, max] over available non-matching FT-AOs, and aggregation yields Figure 5’s median-rank bars.
Loading 2607.23379v1…