PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Autoencoders Identify Without Explaining

Kristina Šekrst, Barbara Geld

arXiv:2610.32691v1cs.CL

TL;DR

The paper argues that successful NLA reconstruction identifies an activation without establishing that its description faithfully explains what the activation does. It examines this distinction and warns that NLA explanations can be treated as testimony even though some internal-processing claims are difficult to verify.

  • Problem

    NLA reconstruction tests whether a description can identify and recreate an activation, leaving open whether it accurately characterizes what that activation does.

    Source: §5 · p. 7, §7 · p. 12

  • Method

    The paper distinguishes a description’s identifying content from its characterizing content and examines NLA examples and proposed checks against that distinction.

    Source: §5 · p. 7, §9 · p. 15

  • Results

    Reconstruction rewards identifying an activation but is insensitive to whether the description correctly characterizes it.

    Source: §5 · p. 7, §9 · p. 15

  • Takeaways & Limitations

    Treating NLA outputs as explanations or testimony can support claims about model thoughts that the reconstruction criterion does not establish.

    Source: §7 · p. 12

    The verbalizer can infer beyond the activation, and false claims about the model’s processing are harder to verify than claims about input context.

    Source: §4 · p. 6

Abstract

from arXiv · show

Natural Language Autoencoders (NLAs) express a model's internal activations in the form of readable text. The verbalizer turns an activation into an English explanation; the reconstructor then converts that explanation back into an activation, and the explanation is considered successful if the round trip brings back the original activation. We claim that this criterion evaluates whether the information is sufficient and does not address descriptive faithfulness. A sentence can contain sufficient information to make it possible to identify an activation even if it wrongly explains what the activation does: in the case of reconstruction, the primary aim is to accurately identify the activations and not to describe them. This issue has the same structure as the chain-of-thought faithfulness problem, in which verbal output that has been trained to go with a process is taken as a report on that process, and it parallels the dissociation between an individual's introspective report and the actual governing process as documented in human psychology. Although proposed as interpretability tools for the purpose of alignment, NLAs reintroduce, through unverified explanations, the epistemic problem that they were intended to resolve.

1 Introduction

The introduction frames interpretability as explaining internal states through independently verifiable tests, then introduces NLAs as systems that translate activations into text and back. It argues that successful reconstruction establishes information sufficiency, not that the text faithfully describes what the activation does.

1 Introduction

  • Interpretability aims to explain what an internal state means, with intervention testing whether changing the state changes the output as the explanation predicts.The explanation and its test are distinct, and the test does not rely on the explanation.Source: §1 · p. 1
  • An autoencoder encodes an input into an internal code and decodes it back, with both parts trained to reduce the difference between input and reconstruction.Restricting the code encourages a concise representation rather than direct copying.Source: §1 · p. 1
  • An NLA uses an activation verbalizer to turn an activation into text and an activation reconstructor to turn that text back into an activation.The two components are jointly trained to reduce reconstruction error; the reconstructor’s result serves as the verbalizer’s reward.Source: §1 · p. 2
  • Sparse autoencoders represent activations as sparse combinations of learned directions, which may correspond to concepts, entities, syntax, or safety-related behaviors.This account relies on the superposition hypothesis that features outnumber dimensions and overlap.Source: §1 · p. 2
  • Successful reconstruction shows that an NLA description contains enough information to recreate an activation, not that it accurately describes the activation’s function.A sentence may identify a state without explaining what that state does.Source: §1 · p. 2

2 Interpretability and the intervention standard

The intervention standard tests an explanation by changing a proposed internal feature and checking whether the output changes as predicted. The paper notes that this standard can rely on a substitute model and requires substantial effort, while sparse-autoencoder interpretability is often assigned after training through predictive labels.

2 Interpretability and the intervention standard

  • Suppressing the Texas feature and substituting a California-related feature changed the answer from Austin to Sacramento, testing the proposed intermediate state.The intervention can refute an explanation if the expected output change does not occur.Source: §2 · p. 2
  • The attribution graph explained a cross-layer transcoder, so its conclusions about the original model depended on the substitute’s faithfulness.The authors compare this reliance to drawing conclusions about an organism from a model organism.Source: §2 · pp. 2–3
  • A transcoder can match the original model’s outputs while its readable components fail to correspond to the original model’s mechanisms.Behavioral matching supports inferences only about what the tested behavior can distinguish.Source: §2 · p. 3
  • The Dallas intervention retains some independence because the original model’s output is observed, but manipulated features still belong to the transcoder.The paper therefore treats it as the best available standard while retaining this limitation.Source: §2 · p. 3
  • Intervention-based interpretability requires substantial human effort per prompt and transcoder training, which the paper says does not scale to production models.The authors distinguish theoretical validity from practical scalability.Source: §2 · p. 3
  • Sparse-autoencoder training rewards reconstruction, not whether learned directions are readable or interpretable.Interpretability is typically assessed afterward by assigning labels based on activation patterns; predictive labels may capture correlation rather than what a direction does, and may fail when data change.Source: §2 · p. 3

3 Natural language autoencoders

The section describes how NLAs replace a learned code with English descriptions that pass between a verbalizer and reconstructor. It then examines the evidence offered for NLA case studies and the risks of treating their outputs as testimony about a model’s internal processes.

3.1 Anthropic research

  • The activation verbalizer converts a layer activation into a short English description, and the activation reconstructor converts that description back into a vector.The description serves as the information transmitted between the two components and replaces the compressed code of a standard autoencoder.Source: §3.1 · p. 4
  • The verbalizer and reconstructor are jointly trained to minimize squared distance between the original activation and its reconstruction, with performance assessed by fraction of variance explained.The reconstructor reads the description in a fixed prompt and maps its resulting activation to the target layer’s activation space.Source: §3.1 · p. 4
  • Warm starts and a KL penalty keep descriptions fluent because reconstruction alone does not require human readability or a semantic link to the activation.The penalty trades off the reinforcement-learning signal against staying close to the verbalizer’s initial writing style.Source: §3.1 · p. 4
  • Evaluation-awareness checks move the NLA measure as expected, but do not establish that the represented state is specifically a belief that the model is being tested.Prompt artificiality or benchmark formatting could also distinguish evaluation from deployment conditions.Source: §3.1 · p. 4
  • The NLA models report FVE of about 0.6 to 0.8, a measure of activation recovery that does not test whether descriptions make true claims about those activations.The reported NLAs each examine one layer, selected approximately two-thirds of the way through the network.Source: §3.1 · p. 4
  • NLA explanations for early Claude Opus 4.6 checkpoints suggested that English-speaking users were being treated as Russian, prompting discovery of mismatched translated fine-tuning data.Other cases included an attribution-graph-confirmed cached-answer explanation and steering that changed outputs in a reward-hacking scenario.Source: §3.1 · p. 5

3.2 From explanation to testimony

  • Anthropic’s public announcement and media coverage described NLAs as revealing model thoughts, although those phrases do not appear in the technical paper.The paper argues that readers may treat NLA outputs as evidence about machine consciousness or moral status.Source: §3.2 · p. 5

4 Alignment and limitations

The section presents NLA applications to alignment but argues that their interpretations remain difficult to verify, especially where ground truth is unavailable. It recommends treating explanations as hypotheses, while noting that this advice depends on independent tests that may be missing in the cases of interest.

4 Alignment and limitations

  • NLAs detect evaluation awareness that changes as expected, even without explicit mention, and an auditing agent identifies a model organism’s root cause without training-data access.The authors regard the latter as evidence that the NLA elicits latent knowledge; previous SAE-based tools had not achieved this without training data.Source: §4 · p. 6
  • The NLA authors acknowledge invented details, harder-to-check claims about processing, and a black-box design that cannot trace explanation clauses to particular activation components.The full language model can make inferences beyond what is contained in the activation.Source: §4 · p. 6
  • Warm-start summaries may introduce bias, and the reconstruction objective can degenerate toward quoting context or producing text that humans cannot read but the reconstructor can invert.The authors describe the warm-start data as inelegant and say such objective pathologies have not yet occurred.Source: §4 · p. 6
  • The authors recommend reading recurring themes as hypotheses and verifying them independently, but such verification is unavailable for some alignment targets lacking ground truth.The section connects this gap to psychology’s finding that coherent verbal reports need not reliably reveal the processes behind a judgment; its human-report framework does not apply directly to NLAs.Source: §4 · pp. 6–7

5 Sufficiency and faithfulness

The section distinguishes reconstruction sufficiency—recovering an activation—from faithfulness—whether an explanation correctly describes the activation’s role. Because reconstruction rewards identifying the state, it can leave true, irrelevant, and false characterizations indistinguishable when they identify the same region.

5 Sufficiency and faithfulness

  • Reconstruction tests whether a description suffices to recover an activation, whereas faithfulness asks whether its attributed role matches the state’s actual role in producing the output.The same sentence can pass the sufficiency test while failing the faithfulness test.Source: §5 · p. 7
  • A clause that identifies a state can give the reconstructor what it needs; an added true or false characterization is credited only for how it moves the reconstructed vector.The Dallas example contrasts clauses that identify a state with characterizations of its role; an intervention can distinguish their predictions, while reconstruction cannot.Source: §5 · p. 7
  • The comparison is framed around the causal facts attributed to a state, not any specific theory of explanation or causation.The authors state that the argument applies across interests, including whether a model follows a hidden goal.Source: §5 · p. 7
  • Figure 2’s idealized descriptions of Texas reconstruct equally well, although only one explains the state’s computational role and the other merely identifies it.The criterion has no means of preferring the role-characterizing description when both receive the same reconstruction score.Source: Figure 2 · p. 8, §5 · p. 8
  • Ablations of claims about input text show reconstruction responds to identifying claims, but those results do not generalize to claims about model cognition.The reported ordering of true and false claims follows proximity to context, and the reconstructor is described as a weak verifier of individual claims.Source: §5 · p. 8
  • The distinction resembles reference versus sense: wording can pick out an activation without saying whether its characterization is true.The paper notes that reconstruction is sensitive to wording and how well it picks out the activation, but not to the truth of what the wording says about it.Source: §5 · p. 9

6 Precedents

The section connects NLA reconstruction and chain-of-thought evaluation to human self-report: fluent or successful verbal output can fail to represent the process it appears to explain. These precedents also show why reports must be evaluated against independent evidence and in light of how they were produced.

6.1 Chain of thought

  • Reconstruction and answer accuracy can both succeed without establishing that an NLA description or chain of thought faithfully represents the process behind it.Chain-of-thought perturbation studies found answers could persist when written steps were disrupted or biased hints went unmentioned; NLA reconstruction likewise does not require accurate self-description.Source: §6.1 · pp. 9–10

6.2 Human introspection

  • Choice-blindness reports can remain coherent after a choice is covertly switched, but independent records reveal the mismatch; misleading feedback also affects reports, especially when internal evidence is weak.Reports may combine decision-related evidence and external feedback, so unreliability in some cases does not establish that all reports are equally unreliable.Source: §6.2 · p. 10
  • Evidence comparing individual reports with group-level effects cannot establish what a particular person could access, though report coherence alone still does not establish accuracy.The section cautions that frameworks for concurrent human verbalization do not transfer directly to NLAs, whose verbalizer describes an already-produced activation.Source: §6.2 · pp. 10–11
  • Psychological research distinguishes reporting attended information from explaining why a decision occurred, with reliability depending on the report procedure (Ericsson and Simon, 1980, 1993).A meta-analysis of 94 studies found no detectable average effect of undirected concurrent verbalization on task accuracy, while requests to describe or explain thinking were associated with performance changes.Source: §6.2 · pp. 10–11

6.3 Self-report as a philosophical problem

  • Introspection may involve interpretation rather than direct access, and an NLA verbalizer likewise interprets a stored vector, making its account a third-person interpretation presented in first-person form.Heterophenomenology treats reports as data about what a subject says or believes, not as proof that the described state is as reported (Dennett, 1991, 2003).Source: §6.3 · pp. 11–12

7 Can a machine be wrong about itself?

A report can identify a state without accurately describing what it does. The paper argues that NLA reconstruction tests identification, while admissible evidence about a state’s purpose also requires reference to that state and independent checks of its claims.

7 Can a machine be wrong about itself?

  • A full encoding may identify a state completely while saying nothing about its function, much as a shirt can identify someone in a crowd without describing their actions.The authors contrast an NLA’s full-encoding objective with a follow-up question and note that sincere reports can still be mistaken about causes.Source: §7 · p. 12
  • Machine self-report is admissible as evidence about its source only when the report is caused by the described state and refers to that state (Šekrst, 2026c).The NLA round trip establishes the causal condition but leaves the reference condition unaddressed; the verbalizer’s self-report vocabulary is installed by training regimes.Source: §7 · p. 12
  • Reconstruction can establish that a sentence identifies an activation, but it does not establish that the sentence’s claims about what the activation does are true.The authors distinguish the identifying part of a description from its characterizing part; reconstruction gives credit for the former but is insensitive to the latter.Source: §7 · p. 12

8 Conditions of admissible explanation

The section proposes conditions for treating NLA characterizations as evidence: checks must be independent and potentially disconfirming, identifying content must be separated from claims about a state’s purpose, and scrutiny should reflect the stakes. These conditions preserve NLAs as hypothesis generators while requiring verification before stronger claims are relied upon.

8.1 Independence

  • An independent explanation check must be capable of refuting the characterization, rather than accepting it as an explanation for anything.The proposed standard tests a sample of counterfactual predictions against model behavior under intervention; because faithfulness is graded, credit tracks the predictions a characterization gets right.Source: §8.1 · p. 13
  • Plausibility and faithfulness can diverge: a human-readable description may be unfaithful, while an invertible private code may reconstruct perfectly without communicating to readers.The section adopts the view that faithfulness comes in degrees.Source: §8.1 · p. 13

8.3 Stakes

  • The required standard rises with the cost of acting on an error; checkable-claim confabulation rates should be reported as a lower bound for claims that are harder to verify.The section cautions that claims about processing may be harder to verify than contextual claims, so the lower bound need not match confabulation rates across claim types.Source: §8.3 · pp. 14–15
  • Fluency can look like accuracy, so identifying and characterizing clauses should be separated, with unchecked characterizations marked as hypotheses.For exploratory use, a reported error rate and separate fields may suffice; for audits, characterizations should pass an intervention check before entering the record.Source: §8.3 · pp. 14–15

8.4 What the conditions do

  • The proposed workflow adds intervention checks after verbalization, routes purpose claims into a separate field, and reports confabulation rates as a lower bound for untested claims.Editing a characterizing clause may help make checks more efficient: if reconstruction is unchanged, the clause lacks score support; if it changes, the difference vector can guide behavioral testing.Source: §8.4 · p. 15
  • NLAs may make the intervention standard more affordable, but the check still has to be performed; the failure is treating a fluent sentence as the final result.The proposed restrictions keep NLAs as generators of hypotheses rather than substitutes for verification.Source: §8.4 · p. 15

9 Conclusion

The conclusion argues that reconstruction identifies which activation a description refers to without validating its claims about that activation. It proposes independent behavioral checks, separate identifying and descriptive content, and confabulation-rate reporting so NLA outputs remain hypotheses whose testing can be assessed.

9 Conclusion

  • Reconstruction cannot certify an explanation because its score rewards identification of an activation but is insensitive to claims about what the activation does.The conclusion also argues that the safeguard of verifying explanations elsewhere is unavailable in some motivating cases, including hidden objectives and strategic deception, where the production model provides no ground truth.Source: §9 · p. 15
  • The proposed restrictions require testing some characterizations under model changes, separating identification from description, and using checkable-claim confabulation rates as a lower bound for uncheckable claims.These restrictions preserve NLAs as hypothesis generators while indicating how well each hypothesis has been tested.Source: §9 · pp. 15–16
  • Fluent NLA descriptions can lead readers to mistake identification of an active state for an explanation of why the model acted.The conclusion compares this misreading to Kempelen’s Mechanical Turk: visible machinery was real but did not explain the chess playing.Source: §9 · pp. 15–16

Related papers

Loading 2610.32691v1…