PaperlayerView exact revision on arXiv

Source-linked AI summary

The Signs Were Always There: Training-Free Concept Detection and Steering in Raw Transformer Dimensions

Varun Reddy Nalagatla

arXiv:2606.12629v3cs.LGcs.AI

TL;DR

Existing feature reading and steering methods typically require learned dictionaries, probes, or labeled data. This paper tests whether raw transformer dimensions suffice, finding that training-free sign-based catalogs detect concepts and steer fluent text across modalities and language models.

  • Problem

    Feature reading and steering typically require trained dictionaries, probes, or labeled data rather than exploiting semantic structure in individual transformer dimensions.

  • Method

    The paper catalogs read signs from hidden-state type caches and write targets from frozen unembedding weights, then applies sign agreement and attention-pathway injection.

  • Results

    62–92% of twelve concepts were steered into fluent text across four language models, while training-free sign catalogs detected 175 categories and scaled to 1500 features.

  • Takeaways & Limitations

    Practical feature reading and steering may use the standard basis, shifting the main challenge from learning rotations to cataloging dimension roles.

  • Takeaways & Limitations

    The evidence covers 4–32B language models and 86M-parameter vision/audio models, leaving persistence at 70B-plus scale open.

Abstract

from arXiv · show

The standard basis of transformer hidden states is a training-free, architecture-general feature basis for detecting concepts and, in language models, steering them; with no learned dictionary. Individual dimensions act as binary registers read one at a time: their signs (+/-1) encode content, their magnitudes strength. A feature is just a subset of dimensions with a consistent sign pattern, read by counting sign agreements. We validate this Bag of Dims (BoD) framework across seven models spanning language, vision, and audio; reading dimensions one at a time loses nothing, as a full-capacity MLP adds zero AUC over per-dim reading. The same per-dimension signs appear in every modality, so they reflect transformer training itself, not the language objective. Sign alone carries predictive content: setting all magnitudes to unity preserves 60-93% top-5 next-token accuracy through the LM head. From a single-token cache (one forward pass per token, no labels) we detect 175 categories at AUC 0.97-0.99 by counting sign agreements, and from random seeds alone discovery scales to 1500 features per model. A trained probe adds only +0.018 AUC and converges to axis-aligned weights: the rotation dictionaries learn buys little. Signs are causally operative: they survive the attention projections, and flipping a concept's sign pattern in the live forward pass suppresses it. Reading and steering are separate roles in the same basis: a concept's reader dimensions are not its writer dimensions. The writer target is just as cheap, the sign of the summed unembedding rows over a few seeds, no training. Injected through the attention output pathway under closed-loop control, it steers concepts into fluent text on four language models (62-92% of twelve concepts). The signs were in the standard basis all along; the open problem is no longer finding the right rotation but cataloging what each dimension encodes.

1 INTRODUCTION

The paper introduces Bag of Dims, a training-free framework that reads semantic features from per-dimension sign patterns and writes concepts through output-projection sign targets. Across language, vision, and audio, these axis-aligned structures are predictive, causally operative, and distinct for reading versus steering.

  • Bag of Dims: Bag of Dims treats each hidden-state dimension as a binary register whose sign encodes content and magnitude encodes strength, with features defined by consistent sign patterns.Reading requires only counting sign agreements, without a learned decoder.
  • Evidence: 80–90% of the time, pure sign agreement narrows a 248K vocabulary to the correct 4096 candidates across three architectures.The result uses Hamming distance and no learned decoder.
  • Evidence: <0.006 bits of pairwise mutual information separates dimension signs, and a full-capacity MLP adds zero AUC over per-dimension reading.Context further lowers cross-dimensional dependence, supporting axis-aligned feature reading.
  • Feature discovery: 175 semantic categories emerge from a single-token type cache at mean per-dim AUC 0.80, while unsupervised discovery scales to 1500 features at 100% yield.A trained probe adds only +0.018 AUC and converges to axis-aligned weights.
  • Causal steering: 62–92% of twelve concepts are steered into fluent text on four language models by injecting a training-free write target through the attention output pathway under closed-loop control.The write target is the sign consensus of output-projection rows over seed tokens and requires no backward pass.
  • Cross-modality universality: 9/12 vision superclasses, 11/12 supervised-vision superclasses, and 50/50 audio categories exceed the stated thresholds, indicating transformer-wide rather than objective-specific structure.The same per-dimension method works across DINOv2, ViT-Base, and AST.

2 METHOD

The method treats transformer dimensions as independent sign channels: features are discovered from single-token hidden-state caches by selecting dimensions with category-separating signs and scored by sign agreement. A separate training-free writer role is derived from output-projection signs to control generation.

  • Raw-dimension representation: A feature is a subset of dimensions with a consistent sign pattern across tokens in a semantic category, readable directly from the standard basis.Signs encode semantic content, while magnitudes encode vote strength.
  • Type-level cache: The type-level cache runs every vocabulary token individually, extracts target-layer hidden states, and stores a V × D matrix without sentences, prompts, or gradients.The cache is computed once per model; sign-only storage uses 1 bit per dimension.
  • Feature discovery: Feature discovery selects 50 single-token anchors, computes polarity-invariant per-dimension AUC, and registers dimensions exceeding τ = 0.75.The resulting prototype is the registered dimensions paired with their expected polarities.
  • Feature scoring: Classification uses a normalized count of sign agreements, equal to 1 − (Hamming distance/|D_c|), with no learned weights.Prototype-level AUC evaluates this composite score, distinct from per-dimension AUC used for dimension selection.
  • Read and write roles: Read roles catalog which categories dimensions detect, whereas write roles measure how changing dimensions drives concept output; the two roles are distinct.The writer target is the sign of summed output-projection rows over seed tokens and requires no backward pass, centering, or training.

3 EXPERIMENTS

Across language, vision, and audio transformers, standard-basis sign patterns preserve predictive information, support training-free feature discovery and contextual reading, and can causally suppress concepts. Reading and production are distinct: sign coalitions detect and suppress concepts, while writer targets derive directly from unembedding weights.

  • Sign-based prediction: 80–90% top-4096 accuracy is achieved by pure Hamming matching with zero learned parameters across all architectures.On Gemma and Mistral, sign matching exceeds the full-dot baseline because it avoids relying on h_final with untied embeddings.
  • Functional independence: 0 AUC is added by a full-capacity MLP over per-dimension logistic regression, establishing that dimensions can be read independently without losing task-relevant information.Mean pairwise mutual information remains below 0.006 bits, and the same result holds for vision and audio transformers.
  • Feature discovery: 175 categories exceed null thresholds on every model, recur across four architectures, and emerge from every random seed as sparse features firing on <0.1% of vocabulary.Randomly discovered features typically show semantic or phonological coherence, and sign detection recovers 23/24 features in a toy superposition model.
  • Contextual reading: 0.814, 0.722, and 0.856 mean contextual AUC are obtained for Qwen, Gemma, and Mistral when type-level prototypes score running text without modification.Approximately 58% of dimensions preserve their type-level signs in context, while the remainder are mostly low-magnitude dimensions.
  • Causal intervention and writing: ∼200–500 coalition dimensions are required before sign flipping suppresses a concept, whereas forcing the same signs toward does not induce it.This establishes the sign coalition as causally operative during generation and distinguishes readout dimensions from the separate production mechanism.
  • Causal intervention and writing: 0 forward or backward passes are required to derive the writer target: summing unembedding rows over concept seeds yields each dimension’s consensus direction.The target specifies which sign maximally boosts the concept’s logit mass and is a property of frozen weights.

4 RELATED WORK

Prior interpretability methods generally recover concepts through learned directions, geometric decompositions, or activation magnitudes. This work instead argues that standard-basis signs provide a sufficient, training-free decoder for detection and steering, while showing that the structure depends on training.

  • Sparse Autoencoders: For 175 semantic categories, standard-basis detection without training achieves comparable performance to sparse-autoencoder feature methods.SAEs decompose hidden states into overcomplete dictionaries and assume features are recoverable through learned rotations; this work reports comparable detection directly from the standard basis.
  • The superposition hypothesis: MI < 0.006 bits in real transformers, versus 0.05–0.10 in Elhage’s toy bottleneck, while sign matching recovers 71–100% of toy features.These results challenge the predicted cross-dimension coupling of geometric superposition and support per-dimension sign matching as a sufficient decoder.
  • Individual neuron interpretability: Residual-stream dimensions show high per-dimension specificity, with AUC 0.80, suggesting polysemanticity may be localized to MLP interiors.This contrasts with the field’s shift away from individual-neuron interpretability after polysemanticity became a consensus view.
  • The privileged-basis question: Unlike prior axis-based and distributed methods that read activation magnitudes or train decoders, this framework tests whether coordinate signs alone decode concepts without transformation.The framework separates sign as content from magnitude and treats subsets of standard-basis dimensions with consistent sign patterns as features.
  • Training-free steering: The write target is sign(P_t W_unembed[t]), requiring no prompt pair, forward pass, or probe, and serves as the same per-dimension sign object used for detection.This contrasts with activation addition and probe-based intervention methods that construct or derive separate steering directions.

5 DISCUSSION

The discussion argues that sign-based feature detection is combinatorially scalable, layer-specific, and compatible with superposition, while separating contextual reading from causal writing. Cross-modal evidence suggests this structure arises from transformer optimization, but its persistence at larger scales and contextual-role catalog remain open.

  • Per-layer coordinate systems: Cross-layer Jaccard overlap is 0.042, indicating that dimension-to-concept mappings are layer-specific rather than shared across the network.Each layer’s FFN rewrites the residual stream into its own coordinate convention.
  • Combinatorial coding: 1500+ features emerge from 2560 dimensions, with ∼897 dimensions per feature and ∼35% overlap, yet 99% fire on <1% of vocabulary.Per-feature sign agreement remains independently detectable, while shared-dimension MI is <0.006 bits.
  • Relationship to the superposition hypothesis: ∼101220 possible sign-pattern features exist for D = 2560, making geometric packing unnecessary as the only mechanism for representing many features.The discussion frames this address space as effectively infinite.
  • Relationship to the superposition hypothesis: 71–100% of toy-autoencoder features are recoverable from per-dimension signs alone, including 23/24 features in a 20-dimensional bottleneck.The toy still exhibits non-orthogonal columns and cross-dimension sign MI of 0.05–0.10 bits, so the results do not refute superposition.
  • Induction requires a separate object from detection: Read-side sign flips suppress target logits but do not induce concepts, because induction uses a nearly disjoint set of write dimensions.This motivates a separate write target for induction.
  • Modality universality: Per-dimension sign structure appears in DINOv2 and AST despite different modalities and training data, suggesting transformer optimization rather than language-specific properties drives it.Whether the structure persists at 70B+ models remains open; tested language models span 4–32B parameters.

6 CONCLUSION … A.2 UNSUPERVISED FEATURE QUALITY

The paper argues that transformer hidden-state dimensions can serve as a training-free basis for detecting and steering semantic features through sign patterns, while read and write roles occupy nearly disjoint dimensions. Comparisons and unsupervised discovery show this approach can achieve high-quality features without the optimization and labeling costs of trained dictionaries.

  • 6 CONCLUSION: Individual dimensions act as binary registers: signs encode semantic features, while magnitudes encode strength.The framework reads dimensions one at a time using sign patterns and magnitudes.
  • 6 CONCLUSION: 62–92% of twelve concepts were steered into fluent generated text across four language models, without training.The write target is injected through the attention output pathway under closed-loop presence control.
  • 6 CONCLUSION: The method requires only a cache and bookkeeping over signs, with sign-pattern capacity of 3D −1 features and no inherent scaling limit as models grow.The paper does not claim the standard basis is unique or optimal, only sufficient for practical feature reading comparable to trained methods.
  • 6 CONCLUSION: Read and write roles occupy nearly disjoint dimension sets, shifting the main challenge from finding rotations to cataloging dimension functions.Read signs detect or suppress concepts, whereas write targets steer concepts.
  • A.1 SAE HEAD-TO-HEAD: The SAE comparison is pinned to Gemma 3-4B layer 25, contrasting 50-anchor sign prototypes using ∼200 dimensions and zero training with 16K trained features.The comparison uses the layer at which Google released Gemma Scope 2 SAE.
  • A.1 SAE HEAD-TO-HEAD: 173/175 categories favored sign prototypes individually, while top-20 SAE features with trained logistic regression reached 0.958 vs 0.952 for sign prototypes.The SAE combination required searching 100 candidate features per category, classifier training, and ∼150 seconds for full-vocabulary encoding versus sub-second sign scoring.
  • A.2 UNSUPERVISED FEATURE QUALITY: The 1500 unsupervised features achieved mean max-score 0.906, median 0.908, with 100% scoring ≥0.80 and 61.4% scoring ≥0.90.Quality was measured on the features’ top-activating tokens without human labels.

B PROBE AND MLP ABLATION

A nonlinear 2-layer MLP with full cross-dimension capacity never improves over logistic regression across training scales, indicating that per-dimension reading captures the available structure. Hard-negative evaluation further yields robust discrimination between semantically adjacent categories.

  • Probe and MLP ablation: The probe is a 2-layer MLP with 128 ReLU hidden neurons, trained with 50–2000 examples per category using an 80/20 split and test AUC.It was designed to learn arbitrary nonlinear combinations of dimensions.
  • Probe and MLP ablation: The MLP never outperforms LogReg on any category or training scale, with consistently negative gaps.Additional capacity is a liability rather than an asset.
  • Probe and MLP ablation: Axis-aligned LogReg weights and an ineffective full-capacity MLP indicate no practical cross-dimension benefit beyond per-dimension reading.The result argues against learned rotation as necessary for concept detection.
  • Probe and MLP ablation: Hard-negative AUC remains 0.92–1.0, with a mean of 0.977, when semantically adjacent categories are used as negatives.The evaluation contrasts animals with food/body parts and weapons with vehicles.

C FULL CATEGORY DATA

The 175 detected categories include ontologically discrete concepts as well as fine-grained, gradient, and context-dependent features, all exceeding null calibration. Categories remain detectable across layers, with separability peaking at h24 and feature density increasing through the stack.

  • Feature categories: 175 categories span strong and weak features, and all exceed null calibration.Strong features are ontologically discrete, whereas weak features include subtypes, degrees, and context-dependent categories.
  • Layerwise separability: AUC ≥0.83 at h0 shows that categories are detectable at all layers, with separability peaking at h24.The number of dimensions exceeding the 0.75 threshold increases progressively through the stack.

D LAYER SWEEP AND PER-HEAD SPECIALIZATION · E RANDOM-INIT CONTROL

Layer sweeps show that feature readability varies substantially across depth and model, while cross-layer overlap is near chance, indicating layer-specific dimension assignments. Randomly initialized controls lack coherent features, supporting training as the source of semantic per-dimension encoding.

  • D LAYER SWEEP AND PER-HEAD SPECIALIZATION: AUC ≥0.84 appears at every Qwen layer, with layer 24 balancing high AUC and moderate dimension count.Layer 32’s larger feature count reflects the prediction transition.
  • D LAYER SWEEP AND PER-HEAD SPECIALIZATION: 87% discoverability at Gemma’s embedding layer falls to 29–39% in middle layers before recovering to 78% at final layer h34.Final-layer mean AUC is 0.772, which is used for Gemma discovery.
  • D LAYER SWEEP AND PER-HEAD SPECIALIZATION: 96–100% discoverability from Mistral h4 onward peaks at mean AUC 0.843 around h20–h28.Mistral’s D = 4096 hidden dimension provides more per-dimension resolution than the 2560-dimensional models.
  • D LAYER SWEEP AND PER-HEAD SPECIALIZATION: Mean cross-layer Jaccard similarity is 0.042, near chance for sparse 2560-dimensional subsets, showing layer-specific dimension assignments.The same physical dimension encodes different categories at different layers.
  • D LAYER SWEEP AND PER-HEAD SPECIALIZATION: At Gemma layer 25, Head 1’s K pathway has 80 feature dimensions concentrated in negation, number, and color.The concentrations are 20 negation, 14 number, and 10 color dimensions.
  • D LAYER SWEEP AND PER-HEAD SPECIALIZATION: At Gemma layer 25, Head 1’s V pathway has 91 feature dimensions concentrated in animal, negation, and emotion.The concentrations are 9 animal, 16 negation, and 14 emotion dimensions.
  • E RANDOM-INIT CONTROL: Randomly initialized models produce no monosemantic, semantically coherent features; residual AUC above 0.50 reflects input-embedding similarity through random projections.The effect is stronger for Mistral’s smaller 32K vocabulary.
  • E RANDOM-INIT CONTROL: Random weights show smooth uniform expansion, whereas trained weights show a pinch and asymmetric internal organization that enables per-dimension feature encoding.This trained-versus-random visualization complements the quantitative random-init control.

F TOY SUPERPOSITION MODEL

In the replicated toy autoencoder, 80–100% of represented features are recoverable from hidden-state sign patterns alone, despite non-orthogonal superposed encoding. Signs carry feature content, while magnitudes refine strength, and cross-dimension coupling increases with bottleneck pressure.

  • Toy superposition: 80–100% of represented features are recoverable from per-dimension sign patterns without a learned decoder or geometric projection.The toy model encodes more features than dimensions, confirming superposition while making sign matching a sufficient decoder.
  • Toy superposition: 242 sign-pattern addresses are available with d = 5, exceeding the 6–10 features represented; at d = 20, the address space reaches 3.5 × 109.The available combinatorial address space makes geometric packing unnecessary in the larger-dimensional regime.
  • Sign-vs-magnitude and cross-dim coupling in the toy: 60–77% of full reconstruction quality is retained by sign-only decoding at d=5, matching real-LM evidence that signs carry content and magnitudes carry strength.The sign/full WMSE ratio is 1.3–1.7×, while real LMs preserve 49–63% top-1 and 72–93% top-5 accuracy with signs alone.
  • Sign-vs-magnitude and cross-dim coupling in the toy: 0.045–0.095 bits of mean pairwise sign mutual information at d=5 is 10–50× higher than real transformers, but falls to 0.005 bits at d=20.Cross-dimension coupling therefore reflects bottleneck pressure rather than a fixed architectural property.

G CROSS-CATEGORY POLYSEMY: CASE DETAIL

Cross-category polysemy is detected by comparing contextual target-token signs with category prototypes, separating strong ontological contrasts more reliably than adjacent concrete senses. Performance is robust to threshold choice in most models, but Qwen3-32B remains near chance across layers, consistent with its low dimension-to-vocabulary ratio.

  • Evaluation: 77 cross-category cases compare the target word’s contextual representation with its category prototype in category-sense and other-sense sentences.Correctness requires the category-sense context to score higher; cases failing target-token verification are excluded, and all remaining cases are reported without selection.
  • Case patterns: Strong ontological pairs such as vehicle, tree, and bird separate cleanly, while failures concentrate in adjacent concrete categories such as body parts and metals.The score is the fraction of prototype dimensions whose contextual sign matches the category polarity.
  • Threshold robustness: 0.768 peak pooled AUC is achieved by Qwen at τ=0.55, versus 0.746 for Mistral at τ=0.55 and 0.738 for Gemma at τ=0.58.Accuracy and AUC remain stable across τ∈[0.55, 0.70] and increase as the prototype broadens with lower τ.
  • Qwen3-32B layer sweep: 0.49–0.71 pooled AUC across Qwen3-32B layers never approaches the 0.72–0.77 achieved by the other models.The weakness is layer-independent rather than a consequence of choosing the wrong readout depth, consistent with Qwen3-32B’s D/V =0.034 limiting per-dimension category sharpness.

H CAUSAL SIGN-FLIP: PER-CATEGORY DETAIL

Live sign-flipping of each category’s prototype suppresses its target-token logits across four language models, while toward-sign and random controls do not. Disjoint-coalition controls show that the causal effect is feature-specific and acts at the coalition level rather than through general competence loss.

  • Causal sign-flip: Away sign-flips suppress every responding category, whereas toward forcing and equal-size random flips do not.The comparison uses the full coalition at τ=0.6 and measures per-category target-logit change.
  • Causal sign-flip: Across four models, flipping a category’s prototype signs during live forward passes suppresses its target-token logits, with 4/5 categories responding strongly in every model.Prototypes are built from type-cache signs, and the intervention flips them at all positions during greedy decoding.
  • Causal sign-flip: 5–24 logits: flipping a concept’s own coalition suppresses it across all four models.The reported causal intervention uses the concept’s full sign coalition rather than an individual dimension.
  • Specificity control: Disjoint coalitions produce effects statistically indistinguishable from random flips, establishing concept-specificity rather than general damage.Although features share ∼35% of dimensions, removing shared dimensions eliminates the apparent cross-concept effect.

I WRITE CATALOG: STEERING DETAIL

The appendix details closed-loop concept steering using sign-correct write targets injected through the attention-output pathway and evaluated against unsteered continuations. Across twelve concepts and four language models, it reports per-concept successes and fixed-prompt generations showing concept-dependent continuations.

  • Evaluation protocol: Each model is scored over 48 trials, covering 12 concepts and 4 held-out prompts, against its own unsteered continuation.A run counts as a success under the appendix’s LLM-judge evaluation protocol.
  • Steering setup: The write target is injected as a sign-agreement coalition into the attention-output input under closed-loop presence control.Runs use sign-correct writing and logit-clean decoding without repetition penalties or n-gram blocking.
  • Per-concept success: Number is weakest on Gemma 2 and Mistral base models, collapsing into digit loops.The reported failures track the presence–coherence frontier rather than broken write targets.
  • Per-concept success: On Qwen3-32B, country and vehicle each reach 0/4 because stronger pushes surface the concepts while breaking sentence coherence.Qwen 3.5-4B is described as the softest overall, including fruit at 1/4.
  • Example generations: With the prompt held fixed, varying only the concept produces different continuations, indicating that the write target drives the output.The examples use the attention pathway, closed-loop control, and logit-clean greedy decoding.

J AXIS-ALIGNED SIGNS IN FFN WEIGHTS AND ACTIVATIONS

Axis-aligned sign structure is present one step before the residual stream, in FFN down-projection columns and neuron activations. FFN activations remain sign-readable despite the nonlinear SwiGLU gate, while single-neuron weight linkage provides only a lower bound on prototype coverage.

  • Interpretation: The measurements are correlational and do not establish that these FFN neurons causally write concepts or serve as the causal writer.The read catalog remains in the residual stream, while steering acts through the attention pathway.
  • FFN weights: 1500 unsupervised prototypes were scored against layer L−1 down-projection columns by prototype-dimension sign agreement after magnitude filtering.The writer layer is L−1 for prototypes at hL; for Gemma, prototypes at h25 use writer layer L24.
  • FFN weights: Single-column agreement cannot reconstruct a full prototype of approximately 900 dimensions and should therefore be interpreted as a lower bound on FFN weight linkage.No single column covers a full prototype, so the observed agreement understates the prototype information carried by the FFN weights.
  • FFN activations: FFN neuron activations provide per-dimension sign features under the same harness used for residual-stream detection.The activations are SwiGLU intermediate SiLU(gate(x)) · up(x), whose sign equals sign(gate(x)) · sign(up(x)); the neuron read hooks block L−1.
  • FFN activations: Within ±0.05 per-dim AUC and ±0.02 at prototype level, neuron activations match the residual stream and exceed null calibration on 165–174 of 175 categories.No random anchor set formed a prototype, with null p99 = 0.500, indicating that sign-readable structure survives the SwiGLU gate.
Loading 2606.12629v3…