PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability

Zhang Enyan, R. Thomas McCoy

arXiv:2608.29034v1cs.CLcs.AI

TL;DR

Interpretability methods provide important but relatively isolated insights into language-model representations, motivating a shared structural account. This paper proposes Tensor Product Representations, derives four methods from them, and finds that constructed variants perform comparably to standard techniques. The resulting account connects these methods through filler-role structure while extending beyond a bag-of-concepts view.

  • Problem

    Interpretability methods reveal partial descriptions of representations, but their shared underlying structure is not obvious.

  • Method

    The paper models representations with Tensor Product Representations and derives interpretability operations through filler-role bindings and unbinding.

  • Results

    The paper derives additive analogies, linear probes, sparse autoencoders, and activation patching from one TPR approximation, with constructed methods performing comparably to standard variants.

  • Takeaways & Limitations

    TPRs connect multiple interpretability findings through a shared structural hypothesis and represent compositional position-content information beyond a bag of concepts.

  • Takeaways & Limitations

    The account is evaluated under the desideratum that representations can be captured by an interpretable closed-form equation and constructed independently of the analyzed network.

Abstract

from arXiv · show

A wide range of methods have been proposed for interpreting language models, delivering important insights into their inner workings. However, different methods and their resulting insights stand in relative isolation: what could the underlying structure of language models be, such that they give rise to all our interpretations? In this work, we propose using Tensor Product Representations (TPRs) as a unifying hypothesis. TPRs give a concrete proposal for how compositional structure could be represented in vector space --- as filler-role bindings. We show, both mathematically and empirically, that TPRs can unify several prior interpretability methods: additive analogies, linear probing, sparse autoencoders, and activation patching. Mathematically, we show that these methods can all be derived from TPRs. Empirically, we apply the derivations to a range of different models --- from small toy models to LLMs --- to construct instances of each of the above interpretability methods; these constructed variants perform comparably to their standard variants. We view this work as a step toward what interpretability will ideally provide: a unified account of the nature of neural networks, corroborated not just by individual observations but also by an explanation of the connections between them.

1 Introduction

The paper asks what underlying structure could unify the partial insights produced by diverse interpretability methods. It proposes Tensor Product Representations as a precise and explanatory account, deriving and empirically constructing several methods from that structure.

  • Interpretability methods reveal consistent but partial descriptions of neural network representations without clearly identifying their shared underlying structure.
  • The paper proposes Tensor Product Representations as a unifying account of neural network representations.TPRs are presented as a formalism for encoding information in vector space.
  • TPRs provide a closed-form approximation of hidden states across varied neural networks and tasks, making the proposed account precise.
  • The paper derives additive analogies, linear probes, sparse autoencoders, and activation patching from TPR operations.
  • Constructed variants of these interpretability methods perform comparably to their standard implementations.
  • Together, the contributions aim to connect multiple interpretability results through a formal theory of representational structure.

2 Background and related work

The background contrasts the limited expressivity of the Linear Representation Hypothesis with TPRs, which encode compositional structure through filler-role bindings. The paper adopts TPRs to explain how diverse interpretability analyses can arise from one representation formalism.

  • Mechanistic interpretability and the Linear Representation Hypothesis: Mechanistic interpretability seeks to reverse engineer neural network representations and algorithms, while the Linear Representation Hypothesis treats information as linearly encoded.
  • Mechanistic interpretability and the Linear Representation Hypothesis: Under the Linear Representation Hypothesis, hidden states are sums of vector embeddings for encoded concepts.
  • Limitations of linear representations: A bag-of-concepts representation cannot distinguish sequences with identical elements in different orders, motivating a structure that systematically represents position-content combinations.
  • Tensor Product Representations: TPRs bind content fillers to structural roles, representing a sequence such as 2,1,7 through its position-specific filler-role pairs.
  • Tensor Product Representations: TPR binding and unbinding are linear or bilinear operations, and unbinding can recover fillers when suitable role vectors are available.
  • Tensor Product Representation hypothesis: The paper uses the TPR hypothesis to provide empirical evidence that neural network representations can be approximated and to derive interpretability methods from that approximation.

3 Approximating neural network representations with TPRs

The paper trains Tensor Product Encoders to approximate frozen neural networks from manually specified filler-role inputs. These approximations achieve strong behavioral and representational agreement on synthetic sequences and substantial variance capture on language and embedding models.

  • A Tensor Product Encoder maps filler-role pairs to a linearly transformed TPR that approximates a neural network representation.
  • TPE training minimizes mean squared error between its output and target hidden states from a frozen encoder-decoder model.
  • The experiments focus on whether TPRs explain representational structure rather than discovering which content is present.
  • Synthetic sequences: 99.97% average substitution accuracy and 0.9385 average R2 were achieved across six synthetic sequence conditions.The conditions combine RNN, GRU, and LSTM encoders with copying and reversing tasks.
  • Synthetic sequences: The synthetic experiments use position and identity as role-filler inputs, with TPEs trained to reconstruct final encoder hidden states.
  • Natural-language representations: TPE approximations reached R2 over 0.90 for all embedding models and over 0.60 for all LLMs on structured subject-verb-object sentences.

4 Additive analogies

The paper shows that additive analogy structure can arise from filler–role bindings in TPRs, both explaining the standard vector operation and enabling comparable or better performance with fewer forward passes.

  • 4.2 TPRs explain additive analogies: TPR structure explains additive analogies because substitutions create cancellable filler–role pair differences.For pairs differing in the same roles, unchanged bindings and bias terms cancel, yielding the standard four-vector analogy.
  • 4.3 Additive analogies can be constructed from TPEs: TPE-constructed analogies match or outperform standard vector analogies across synthetic sequences and SVO sentences.The TPE computes the analogy offset directly without embeddings for B and C.
  • 4.3 Additive analogies can be constructed from TPEs: TPE-constructed analogies require one forward pass instead of three to compute the analogy’s left-hand side.The offset is computed from the trained TPE and then subtracted from A’s embedding.
  • 4.3 Additive analogies can be constructed from TPEs: 0.94 ± 0.01 cosine similarity separates TPE binding differences from 0.89 ± 0.03 for offsets between analogous ModernBERT sentence pairs.The PCA visualization shows that TPE-predicted differences closely align with observed sentence-triplet differences.

5 Linear probing

The paper interprets linear probes as TPR unbinding operations and constructs them from trained TPEs. Constructed probes largely match trained-probe accuracy across sequence and SVO tasks while exposing shared filler representations across roles.

  • 5.2 TPRs explain linear probing: A linear probe can be constructed from a TPE by recovering the TPR, unbinding a role, and projecting the filler onto its vocabulary.These three operations are linear and compose into explicit probe weights.
  • 5 Linear probing: Constructed probes largely match trained-probe accuracy across digit sequences and SVO sentences.They achieve near-perfect accuracy where trained probes perform well and similarly lower accuracy where trained probes perform worse.
  • 5.2 TPRs explain linear probing: Reusing filler embeddings across positional probes suggests that different probes decode shared structural components while unbinding different roles.The relationship between subject and object probes is explained through shared filler representations.

6 Sparse autoencoders

The paper derives sparse autoencoders from TPR unbinding by assigning features to filler–role pairs. These constructed SAEs reconstruct representations well, although trained SAEs achieve higher reconstruction scores and supervised variants provide stronger feature quality.

  • 6.1 Performing SAE experiments for our setups: Both top-k and supervised SAEs perform well in reconstruction, while supervised SAEs also achieve high feature quality.Top-k SAEs have lower feature quality, as they are not supervised toward specified concepts.
  • 6.2 SAEs can be derived from TPRs: TPR-derived SAE encoders use one feature for each filler–role pair, with the decoder set to the encoder’s pseudoinverse.The encoder weights and biases are obtained by applying the TPR-derived construction to each pair.
  • 6.2 SAEs can be derived from TPRs: Constructed SAEs achieve average R2 = 0.96 reconstruction across embedding-model and language-model representations.Their reconstruction score is lower than that of trained counterparts.
  • 6.2 SAEs can be derived from TPRs: The TPR-derived reconstruction gap may reflect the restriction that filler–role pairs are represented as present or absent.The paper contrasts this constraint with the greater expressivity of trained SAEs.

7 Activation patching

The paper constructs activation patching interventions from TPEs by representing source–destination differences as filler–role binding differences. Across OLMo-2-13B layers and token positions, TPE-constructed patching produces near-identical restoration results to standard patching.

  • 7.1 Motivation: Activation patching is hypothesized to modify TPR filler–role binding structure, making causal intervention outcomes a test of whether that structure affects model behavior.The paper frames successful TPE-constructed interventions as evidence that the identified TPR structure is causally implicated in behavior.
  • 7.2 Standard activation patching: The experiment uses token-level activation patching on OLMo-2-13B, replacing a destination activation with the analogous source activation at a selected layer and token.If the activation contributes to the output, the model’s response should change toward the response elicited by the source prompt.
  • 7.3 Activation patching can be constructed via TPEs: TPEs construct destination activations by predicting the source–destination activation difference instead of running another language-model forward pass.The method is applied across all token positions and layers, with restoration scores compared against standard activation patching.
  • 7.3 Activation patching can be constructed via TPEs: TPE-constructed patching achieves near-identical results to standard activation patching, reproducing both effect magnitude and information-position shifts.Figure 6 compares restoration scores across layers and 14 prompt token positions on OLMo-2-13B.
  • 7.4 Broader connections: The TPE construction connects activation patching to intervention methods that assume adding or subtracting representation vectors can causally change model behavior.The paper relates this assumption to steering vectors and causal abstraction.

8 Discussion

The discussion presents TPRs as a structural hypothesis that connects multiple interpretability methods and extends beyond one-dimensional linear representations. It argues that such structural accounts can unify findings across models and tasks while supporting broader descriptions of language-model representations.

  • Unified interpretability structure: A single TPR structure explains additive analogies, linear probes, sparse autoencoders, and activation patching through distinct operations on shared representations.The paper interprets vector differences as binding differences, probes as role unbinding, SAE dictionaries as internally structured, and patched-activation differences as filler–role differences.
  • Structural insights: Structural assumptions can connect interpretability results across models and tasks by allowing multiple analyses to be derived from the same TPE approximation.The discussion treats successful experiments based on shared structural assumptions as evidence supporting those assumptions.
  • Long-term goals: The paper envisions interpretability developing structural descriptions that persist across individual behaviors and architectures.This longer-term goal is framed as a broad science of language models focused on the nature of neural networks across model classes.
  • Relationship to the Linear Representation Hypothesis: TPRs extend the Linear Representation Hypothesis by explaining how position and content are compositionally stored together in a single activation.The paper contrasts filler–role bindings with a bag-of-concepts view and reports generalization to held-out filler–role pairs.
  • Relationship to the Linear Representation Hypothesis: TPRs are multi-dimensionally linear and can capture structures that one-dimensional linear probes cannot fully recover.For the reported GRU and LSTM sequence representations, TPEs achieve perfect substitution accuracy despite incomplete recovery by linear probes.

9 Conclusion

The paper concludes that TPRs unify four interpretability methods and advance a formal account of shared representational structure. It also identifies extensions to additional methods and more complex structures as future work.

  • 9 Conclusion: TPRs unify four popular interpretability methods under a shared formalism.The paper connects additive analogies, linear probes, sparse autoencoders, and activation patching through TPR operations.
  • 9 Conclusion: Future work includes extending the unification to more methods and tasks and representing structures beyond the expressivity of the Linear Representation Hypothesis.The paper specifically mentions complex, varying structures such as syntax trees.
  • C Recovering a TPR from neural network hidden state: The authors use regularized pseudoinverses because naive inversion can be unstable for noisy or rank-deficient projection weights.They report that Tikhonov regularization with λ = 0.1 worked well in their experiments, while output-layer regularization was tuned separately.
  • C Recovering a TPR from neural network hidden state: Recovering TPRs can require an ill-conditioned pseudoinverse when projection weights down-project the representation.In one example, a 256-dimensional hidden state approximates a flattened 4096-element TPR.
  • D Constructing linear probes from TPRs: The construction of analytic probes recovers a TPR, unbinds a role, and projects the filler onto the vocabulary without additional training.This yields a fully specified linear probe from the TPR approximation.

F.1 Task and dataset

The sequence-to-sequence experiments use synthetic copy and reverse tasks with one-layer recurrent models, then reuse the trained checkpoints for TPE, analogy, and probe experiments. Evaluation includes token- and sequence-level accuracy, TPE reconstruction, and probe accuracy.

  • F.1 Task and dataset: The dataset contains 50,000 length-6 examples for each synthetic copy or reverse task.It uses a 20-token vocabulary, with tokens sampled independently and outputs generated algorithmically.
  • F.2 Sequence-to-sequence models: One-layer RNNs, GRUs, and LSTMs are trained on both tasks and reused across TPE approximation, linear-probe, and analogy experiments.The recurrent models use embedding size 64 and hidden state size 256.
  • F.2 Sequence-to-sequence models: The models are evaluated with teacher-forced token accuracy and free-running sequence accuracy on the test split.Sequence generation ends at <eos> or after 10 tokens.
  • F.3 Tensor Product Encoders: TPEs reconstruct recurrent hidden states and are evaluated using R2 and decoder substitution accuracy.The TPEs are stored and reused for analogy evaluation and probe construction.
  • F.4 Linear probes: Six position-specific probes are trained for each of six sequence-to-sequence models, and test accuracy is reported against analytically constructed probes.The comparison is summarized in Figure 5 and Table 5.

G The SVO sentences task

The SVO experiments represent sentences through subject, verb, and object filler-role pairs across embedding models and decoder-only language models. TPEs and linear probes are evaluated on held-out sentence data, with trained probes generally performing near perfectly.

  • G.1 The SVO sentences task: The SVO dataset exhaustively combines 77 occupation nouns with 5 verbs into 29,645 sentences.Sentences follow the template “the <subject> will <verb> the <object>.”
  • G.2 Tensor Product Encoders: Each sentence is encoded as three filler-role pairs corresponding to its subject, verb, and object.For example, “the carpenter will see the magician” becomes (carpenter, subject), (see, verb), and (magician, object).
  • G.2 Tensor Product Encoders: TPEs are trained for three embedding models and three language models using held-out evaluation and test splits.The LLM representations are final-layer hidden states at the sentence-final period token.
  • G.3 Linear probes: All trained probes perform well, and every probe except the ModernBERT object probe reaches near-perfect accuracy.Each role probe predicts either a 77-way subject/object label or a 5-way verb label.

H Additive analogy construction and evaluation

The appendix constructs analogy quartets by controlled role substitutions and evaluates standard versus TPE-derived analogy vectors with top-k accuracy. It also describes SAE variants and measures reconstruction and feature quality.

  • H Additive analogy construction and evaluation: Sequence analogy quartets vary one to three positions while constructing A, B, C, and D from controlled substitutions.The evaluation samples 1,000 analogies per model and task.
  • H Additive analogy construction and evaluation: SVO analogy quartets vary exactly one role while holding the other two fixed.The construction yields 44,468 analogies, including 22,488 subject analogies and 21,980 object analogies.
  • H.3 Evaluation: Analogy vectors use either the standard e(A) − e(B) + e(C) form or a TPE-constructed variant, ranked against candidate embeddings.Top-k accuracy is the fraction of analogies whose target rank is at most k.
  • H.3 Evaluation: Top-k accuracy is computed from cosine similarities between analogy vectors and candidate embeddings.The vectorized implementation normalizes embeddings, indexes each target similarity, and counts candidates with greater similarity.
  • I Training and evaluating sparse autoencoders: The SAE experiments compare top-k SAEs with supervised SAEs on sentence representations from embedding models and language models.They report reconstruction R2 and feature quality on held-out test data.
  • I Training and evaluating sparse autoencoders: Supervised SAEs assign 159 features to subject, verb, and object concepts and add a supervision loss.The 159 features comprise 77 subjects, 77 objects, and 5 verbs.

J.1 Synthetic Sequences

The experiments test whether Tensor Product Encoders generalize beyond observed filler-role bindings in synthetic sequences and SVO sentence embeddings. They retain strong reconstruction and substitution performance on held-out combinations.

  • Synthetic sequences: Synthetic copy and reverse datasets withhold six number-position pairs from training, validation, and test splits.The generalization split contains 5,000 examples, each with at least one held-out pair.
  • Synthetic sequences: Digits TPEs average R2 = 0.9287 and substitution accuracy 0.9404 on the held-out generalization split.RNN TPEs achieve perfect accuracy on test and generalization splits, while GRU and LSTM TPEs remain at least 90% accurate out of distribution.
  • SVO sentence embeddings: SVO experiments withhold distinct noun sets in subject and object roles to test generalization across unseen filler-role combinations.Verbs occur only in the verb role, so the holdout targets noun-role pairs.
  • SVO sentence embeddings: SVO TPE reconstruction reaches average R2 = 0.9514 on generalization, compared with 0.9727 on the in-distribution test split.The result indicates that filler-role structure transfers to systematic substitutions.
Loading 2608.29034v1…