Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures

1 Introduction

• The motivation is to combine latent input-feature extraction with output-oriented inspection of internal-state and textual-output misalignment, also called Eliciting Latent Knowledge.The paper focuses on target-concept retrieval and compositional-concept extraction as latent dynamics related to downstream behavior.Source: §1 · pp. 1–2
• Hyperdimensional Probe combines bounded, interpretable supervised features with a queryable VSA proxy space, supporting feature sets beyond single-token output representations.Its unidirectional embedding-to-VSA transformation does not permit causal validation in the core implementation.Source: §1 · p. 2
• The two experimental settings examine concept compositionality in analogy-style input completion and concept dynamics before and after QA text generation.The completion tasks ask whether models represent in-context concepts, key–value associations, or output concepts, and whether findings vary by input type and model.Source: §1 · p. 3
• The method avoids logit analyses’ vocabulary constraints and reports cleaner results than SAE-based analyses in the paper’s bounded conceptual-feature setting.It also supports probing the full residual stream without the layer-selection stage common in earlier probing work.Source: §1 · p. 3

2 Related work

• Supervised probes map embeddings to task-relevant input features, but task-specific training can limit generalizability and complicate separating decoded information from probe-induced learning.They are trained directly on targeted features, ranging from syntax to coordinates and truthfulness.Source: §2 · p. 3
• SAEs learn proxy spaces through sparse dictionary learning, but their unbounded features can be difficult to control, name, align, and associate with an optimal hidden layer.Feature naming can depend on output-vocabulary labels or ambiguous, data-dependent descriptors.Source: §2 · p. 3
• Logit attribution projects embeddings into the output vocabulary, limiting analysis to token-aligned features and constraining higher-level semantic feature inspection.The cited approaches include unembedding-matrix projection, learned affine transformations, and patched inference.Source: §2 · pp. 3–4
• The proposed framework avoids SAE feature naming and optimal-layer selection while combining input-oriented feature extraction with output-oriented analysis.It learns a transformation from embeddings to a controlled proxy space rather than predicting latent features directly.Source: §2 · p. 4
• Hyperdimensional Probe integrates VSA symbolic representations into supervised probing, bridging supervised probes and sparse representation learning rather than offering another conventional probe or SAE variant.Its approach uses a bounded compositional feature space, a learned sparse proxy space, and features that can vary in abstraction, cardinality, and symbolic origin.Source: §2 · p. 4

3 Background

• VSAs map concepts or data structures to random points in high-dimensional space, where concentration of measure supports exponentially many nearly orthogonal concept vectors.A codebook maps predefined concepts to hypervectors that can be composed using orthogonality and hypervector operations.Source: §3 · p. 4
• The paper adopts MAP-Bipolar, whose codebook stores bipolar random vectors for atomic concepts and supports comparing representations with known vectors.The codebook vectors are generated deterministically from seeds to ensure orthogonality and independence.Source: §3 · p. 4
• Binding associates features with values, while bundling combines contextualized concepts into a vector similar to its constituents, enabling retrieval.The paper gives (USA ⊙ dollar) as a binding example and (USA + Mexico) as a bundling example.Source: §3 · p. 4
• In MAP-B, unbinding recovers an elemental vector from a binding by multiplying by the other vector’s inverse, which is itself in this architecture.Bundling irreversibly blends its components, with similarity to the originals diminishing as their number increases.Source: §3 · p. 5

4.1 Synthetic corpus

• The corpus combines Google Analogy Test Set and BATS pairs spanning 44 domains in five categories, plus mathematical analogies.The resulting analogy pairs support controlled tests of factual, semantic, morphological, and mathematical relations.Source: §4.1 · p. 5

4.2 Training examples

• Each input is encoded by binding keys to values and bundling the resulting associations; the codebook contains 2,996 unique concepts in 4,096-dimensional hypervectors.The codebook uses all unique corpus words, and the average pairwise cosine similarity is 0 ± 0.02.Source: §4.2 · p. 6

4.3 Processing neural embeddings F

• The pipeline removes target b2, extracts final-token residual-stream embeddings from the latter half of layers, then reduces them with k-means clustering and sum pooling.The chosen cluster count is k = 5; the paper notes adjacent layers are highly correlated and compression lowers computational cost.Source: §4.3 · p. 6

4.4 Neural VSA encoder T

• A supervised three-layer MLP maps LLM embeddings to known bipolar VSA encodings, learning latent features with explicit semantics.The encoder has 55M–71M parameters depending on input embedding size and uses BCE loss with an MSE regularization term.Source: §4.4 · p. 7
• 0.89 test-set average cosine similarity and 0.94 average binary accuracy show alignment between predicted and target VSA encodings across tested models.The 0.89 cosine similarity is reported for Llama 3.1-8B; the paper says layer count had no effect and lower embedding dimensions were slightly detrimental.Source: §4.4 · p. 7

4.5 Probing VSA encodings I

• Concept retrieval dynamically tests candidate unbindings, using cosine similarity and a 0.1 detection threshold to isolate concepts from encoded associations.Across models, 80% of unbinding operations used the target pair’s key, while 12% required no unbinding.Source: §4.5 · p. 8

5.1 Experimental setup

• Held-out verbose templates shift decoding from the training colon to `to`, testing the encoder on syntax and embeddings not seen during training.Evaluation measures LLM next-token prediction and VSA target retrieval using precision@k.Source: §5.1 · pp. 8–9

5.2 Concept extraction

• 83% average probing@1 across models recovered target concepts; Key | Target appeared in 60% of GPT-2 and 85% of Llama 3.1 cases.The target-retrieval results are shown alongside model next-token performance and logit- and SAE-based analyses.Source: Figure 4 · p. 9, §5.2 · p. 10

5.4 Comparison with Sparse Autoencoders

• 46% average precision@10 across models; SAE retrieval matched VSA probing@1 only when considering the top-100 activated neurons.The comparison is reported with target-retrieval results in Figure 4 and Table 8.Source: Figure 4 · p. 9, §5.4 · p. 12

5.2 Concept extraction

• Blank extraction ranged from 1–1.8% for linguistic analogies and 5.3–7% for factual or semantic relations to 34.8% for hierarchies and 89.5% for mathematics.The authors suggest the high mathematical blank rate may reflect the rarity of numerical analogy tasks relative to equation solving and standard math problems in training data.Source: §5.2 · p. 10
• 8% precision@1 for 109B Llama 4 Scout was the lowest, despite probing@1 ranking among the best; prompt design and tokenization may contribute to the gap.The model often predicted a space instead of the target word, which still frequently appeared in its top five predictions.Source: §5.2 · p. 10

5.3 Comparison with Logit Attribution

• DLA produced no concepts in nearly 30% of analogies, 17% more than VSA; VSA recovered key–target pairs in 57% of DLA-empty cases.The authors also report greater variability and lower target-retrieval performance for DLA, while VSA remained robust under the same zero-shot prompting setting.Source: §5.2 · p. 10, §5.3 · p. 11

5.4 Comparison with Sparse Autoencoders

• SAEs were trained per model with latent width four times the embedding size and 2% top-k sparsity, then tested on syntactically shifted inputs.The encoder used a linear layer with ReLU and the decoder a linear layer; test-set reconstruction cosine similarity averaged 0.98. Features were named using logit-based top-10 token matching.Source: §5.4 · pp. 11–12
• 43% of top-10 SAE outputs were out-of-context-only, while about 32% combined key and target concepts with additional out-of-context concepts.Without task-specific relevance filtering, SAE features may include unrelated concepts; the choice of k also affects evaluation and complicates comparison with VSA probing.Source: §5.4 · p. 12, Table 4 · p. 12

• The two settings use fundamentally different proxy-space construction strategies, with the QA encoder learning lexical-semantic features for a variable number of concepts.Source: §6 · p. 13
• The QA analysis uses SQuAD questions and answers to ground lexical-semantic concepts, then compares extracted concepts with question–answer concepts by cosine similarity.The setting evaluates 10,000 sampled inputs before and after generation; the probe extracts an average of three concepts at both token-generation stages.Source: §6 · p. 13
• The Llama QA encoder maps embeddings to VSA encodings with test-set cosine similarity 0.53 and binary accuracy 0.74, versus null-model cosine similarity 0.24.Source: §6 · p. 13
• Question-related concept similarity fell 4.8% overall and 8.0% in Llama’s error subset, while answer-related similarity was unchanged overall and rose 3.2–3.5% in errors.Figure 5 distinguishes the full sample from failure instances; a weak positive correlation links post-generation question-concept proportion with token-based F1 (Spearman 0.2, p-value 1e−99).Source: §6 · pp. 13–14, Figure 5 · p. 14
• In the laboratory example, Llama loses the question concept hydrogen during generation while refining answer concepts, then outputs “water and heat” (F1 = 0.57).Source: §6 · p. 14
• For GPT-2, semantic-oriented similarity fell 13.7% for question concepts and 4.5% for answer concepts after generation, but no concept-based difference appeared in its error subset.The authors associate GPT-2’s errors with shallow, input-linked concepts rather than Llama’s shifted focus.Source: §6 · p. 14

• VSA probing provides empirical evidence that LLM embedding features can be faithfully captured and supports unified analysis of features aligned with both inputs and outputs.Source: §7 · p. 14
• The method combines supervised probes’ top-down interpretability, SAE-style sparse proxy spaces, and output-oriented investigation, while avoiding layer selection through its ingestion algorithm.Source: §7 · p. 14
• The experiments report differences in concept compositionality across models and input types, and associate Llama errors with lost question focus and GPT-2 errors with shallower QA representations.Source: §7 · p. 15
• The approach applies to autoregressive and Hugging Face models; VSA and hypervector algebra are also presented as promising for steering generation and studying other NLP tasks and multimodal features.Source: §7 · p. 15
• The current unidirectional transformation prevents causal validation, although the paper reports bidirectional-design experiments for steering text generation.Source: §7 · p. 15
• The authors could not measure the effectiveness of their strategies for mitigating probe-induced learning confounds.Source: §7 · p. 15
• VSA-based probing still requires a predefined set of latent features, despite avoiding reliance on the LLM’s output vocabulary.Source: §7 · p. 15

• Source code and synthetic corpora are slated for release upon acceptance, and a README.md provides reproduction instructions.Source: Reproducibility statement · p. 15
• The paper directs readers to supplementary sections for pipeline, training, model, ingestion-algorithm, and computational-cost details.Source: Reproducibility statement · p. 15

A Limitations

• The effectiveness of data augmentation and syntactically diverse test inputs for mitigating confounds could not be measured.Source: A Limitations · p. 20
• Key-value association artifacts occurred in 2% of 114,099 textual inputs, and the authors used randomized-embedding and unrelated-concept controls to investigate probe-learning confounds.Source: A Limitations · p. 20
• Although VSA probing avoids dependence on the LLM’s output vocabulary, it still requires a predefined set of concepts.Source: A Limitations · p. 20

B Algorithm to process neural embeddings of LLMs

• The ingestion procedure returns a compressed state for next-token prediction by retrieving the residual stream and retaining last-token embeddings from the bottom half of the layers.Source: B Algorithm to process neural… · p. 20

C Architecture of our Hyperdimensional probe

• The encoder configuration is specified for an input embedding dimension d.The supplied table caption does not provide the configuration details.Source: Table 5 · p. 21

D Training performance of the neural VSA encoders

• The encoder’s test performance is reported across models, with a null model using randomly permuted embeddings as the baseline.The supplied table caption does not include the performance values.Source: Table 6 · p. 22

D.1 Training details

• The encoder was trained for 421 epochs on average, using early stopping, AdamW, and dynamically binarized outputs.The average learning rate was approximately 3e−5; the effective batch size increased at epochs 110, 310, and 410.Source: §D.1 · p. 22

D.2 Hugging Face repositories for the considered LLMs

• The considered models are Llama 4 Scout, Llama 3.1, Phi-4, Pythia, OLMo-2, and GPT-2.The appendix lists the corresponding Hugging Face repositories.Source: §D.2 · p. 22

E Unbinding stage from Section 4.5

• Four unbinding strategies account for 97% of operations across models: Key, NONE, Example, and Context.The table also distinguishes Greedy selection, which draws a candidate from the vocabulary rather than the input.Source: Table 7 · p. 23

F.1 Validation strategy

• Both controls yielded very low precision: irrelevant concepts produced meaningless outputs, and permuted embeddings produced poor results.The controls used unrelated concepts and randomly permuted LLM embeddings.Source: §F.1 · pp. 23–24
• Probing cannot provide absolute certainty because human-interpretable information in LLM embeddings is not explicitly known.The authors additionally evaluated on textual inputs distinct from training inputs.Source: §F.1 · p. 24

F.2 Diagnosing erroneous answers from Llama 4

• 87% probing@1 for Llama 4 accompanies whitespace-token outputs on 76% of the corpus, which the authors suggest reflects syntactical handling rather than absent analogical reasoning.The whitespace rate compares with an 8% average for other models and 30% for Llama 3.1.Source: §F.2 · p. 24

G Experimental comparison

• Direct Logit Attribution offers a direct comparison without feature naming, but its output is constrained to the model’s vocabulary.SAE comparisons require feature naming and filtering, making them less direct; the methods also differ in their operating principles.Source: G Experimental comparison · p. 25
• In bounded conceptual frameworks, SAE analyses yielded less structured findings and greater noise than VSA probing.The authors attribute the difference to SAEs’ bottom-up, unsupervised feature discovery.Source: G Experimental comparison · p. 25

G.1 DLA-based experimental results

• DLA yields no concepts in nearly 30% of analogies on average, while VSA extracts the key–target pair in 57% of all analogies where DLA yields none.In that subset, VSA returns no concepts in 28%; DLA yields the target with its key in 26% of cases.Source: §G.1 · p. 25
• The DLA comparison uses fuzzy token-to-concept matching and projected next-token predictions from middle-to-last layers at the last token.The full-corpus table covers the same middle-to-bottom hidden-layer range and highlights common extracted concepts.Source: §G.1 · p. 25, Table 9 · p. 26
• Across models, VSA fails to retrieve concepts in 43% of these cases for Llama 4, versus 14% for Llama 3.1.For OLMo-2, linguistic analogies have the highest reported Context | Target retrieval rates, while mathematical analogies show nearly no concept retrieval (91%).Source: §G.1 · p. 26

G.2 Raw results obtained though the DLA probing technique

• The supplied DLA raw-output figure covers OLMo-2 for one sampled analogy; the section also identifies an SAE training-performance comparison against a permuted-embedding baseline.The table captions do not provide the SAE performance values.Source: Figure 6 · p. 27, Table 12 · p. 28

G.3 Concepts extracted by DLA when VSA yields no concepts

• When VSA yields no concepts, DLA also yields none in a majority of instances: 59 ± 15%.The table caption notes high variability among models.Source: Table 11 · p. 28

I.1 Generalization of input representation

• VSA encodings use binding and bundling to combine hypervectors for input words and key–value pairs.The authors note that input-feature extraction can require human expertise for tasks such as toxicity detection; after features are extracted, encoding is automatic, efficient, and scalable.Source: §I.1 · p. 30

I.2 Applicability to other downstream tasks

• VSA-based probing is presented as a flexible framework for examining how LLMs encode and relate features, from syntactic structures to concepts such as gender bias or toxic language.Potential applications include linguistic feature extraction, toxicity detection, bias classification, QA templates, and inputs combining text with other modalities.Source: §I.2 · p. 30

I.3 Handling of Out-of-Domain Data

• The encoder requires examples of the target latent features and cannot extract concepts it has never encountered.The authors report evaluating on different textual inputs that share semantic concepts but differ syntactically; effectiveness also depends on whether the targeted feature is likely encoded in the LLM’s vector space.Source: §I.3 · pp. 30–31

I.4 Applicability for steering text generation

• The preliminary steering experiments achieved 0.99 average test-set binary accuracy for VSA concepts and 0.96 cosine similarity when reconstructing LLM embeddings.The method projects concepts back into the LLM representation space, then subtracts the source-language concept and adds the target-language feature; the authors report steering a French input toward Spanish.Source: §I.4 · p. 31, Table 13 · p. 31

J Question-answering setting from Section 6

• The QA setup probes the model’s last-token state before and after generation, comparing its representations with the codebook to extract concepts.The appendix describes SQuAD question inputs with contextual text and a corpus of 693,886 training examples.Source: J Question-answering setting from… · p. 32

M Synthetic corpus

• The synthetic corpus appendix reports domain clustering and augmented-domain cardinalities, without supplying their values in the provided captions.It also identifies a table of knowledge bases used to construct the corpus.Source: Table 14 · p. 37, Table 15 · p. 37, Table 16 · p. 37

O.2 Analysis of representation redundancy

• For OLMo-2, the residual-stream Gram-matrix spectrum has about 3–4 dominant modes; the first normalized eigenvalue is 0.65 versus 0.17 for the second.The authors describe this as an approximately low-rank embedding space and hypothesize that the leading component may correspond to next-token prediction representations.Source: §O.2 · p. 38, Table 17 · p. 39

O.3 Silhouette analysis for determining optimal range of clusters

• The clustering analysis reports silhouette scores across cluster counts and the distribution of layer-cluster assignments for Llama3.1-8B.The captions specify a random sample of 10,000 textual inputs for the silhouette analysis but do not provide score values.Source: Figure 14 · p. 39, Figure 15 · p. 40

O.5 Ablation study on the dimensionality-reduction steps

• 742 million trainable parameters without either compression step, versus 71M after ingestion reduction—a tenfold increase that raises training and inference footprint.The uncompressed residual stream becomes a flat input of dimension 168960; the passage also describes lazy extraction in an approximately 10^5-dimensional, low-rank space as computationally inefficient.Source: §O.5 · p. 40
• Removing sum pooling is expected to raise encoder cost to 155M trainable parameters, or 2x, while other dimensionality-reduction approaches may also work.The passage says this single-step removal should increase computational cost rather than affect probe outputs.Source: §O.5 · p. 40

P Proof of concept for hyperdimensional probe in multimodal settings

• Figure 16 illustrates a MNIST-based mathematical-analogy probing procedure and VSA encodings of multimodal inputs with textual and image features.Source: Figure 16 · p. 41

Q Computational workload

• LLM inference was the most demanding stage, while encoder training used a single GPU and probing's vector multiplications were negligible.The authors report testing six models from 355M to 109B parameters; encoder training took about eight hours per model, and probing Llama 3.1-8B embeddings used around 95 GPU hours, at roughly three seconds per instance.Source: Q Computational workload · p. 41