Identifying Linear Relational Concepts in Large Language Models

• Identifying concept directions can help analyze how concepts are created and changed during inference and investigate incorrect or undesirable model responses.Source: §1 · p. 1
• Probe classifiers may capture features correlated with a concept rather than its causally influential direction, and high-dimensional hidden states may require many training samples.Source: §1 · pp. 1–2
• LRCs outperform standard probing classifiers such as linear SVMs in classification accuracy and causality, with high scores across four concept types.Source: §1 · p. 2
• LRCs can edit model output: subtracting the “Located in England” direction from London’s activation and adding “Located in France” can produce “London is located in France.”Source: §1 · p. 2
• The method extends LREs to multi-token objects, uses non-terminal object layers, and inverts LREs to derive subject-space concept directions.Source: §1 · p. 2

2 Background

• Prior work describes features as directions in hidden activation space and mid-level transformer MLP layers as key-value stores that enhance subject tokens with factual-relation information.Source: §2 · p. 2

2.1 Linear Relational Embeddings

• An LRE models a relation as an affine linear transformation from subject activations to object activations, with its parameters estimated from relation examples.Source: §2.1 · p. 2
• Faithfulness evaluation restricts LREs to a single object token at the final layer, so objects sharing an initial token can be indistinguishable.Source: §2.1 · p. 3
• A low-rank pseudo-inverse maps object changes into subject edits, and the paper builds on this inversion to target subject activations rather than object activations.Source: §2.1 · p. 3

• An LRC is a unit-length direction for a relation–object concept, evaluated against the final-token hidden state of a subject in an autoregressive model.Source: §3 · pp. 3–4
• Because the LRE is an intermediate step rather than a directly faithfulness-evaluated predictor, object activations can come from earlier model layers.Source: §3 · p. 3
• Training prompts are balanced across possible objects, and examples the model answers incorrectly are discarded as likely noise for the target concept.Source: §3 · p. 3
• The LRE uses the Jacobian of the mean activation across all object tokens, allowing multi-token objects to be modeled rather than only their first token.Source: §3 · pp. 3–4
• The method applies a low-rank pseudo-inverse to map object activations back to subject activations, averages the resulting vectors for each relation–object pair, and normalizes the direction.Source: §3 · p. 4

4 Results

• The evaluation uses the relations dataset and compares LRC against SVM, averaged activation directions, and a final-object-layer LRC variant.The study evaluates Llama2-7b and GPT-J, and measures classification accuracy and causality.Source: §4 · p. 4, §4.1 · pp. 5–6
• The evaluation excludes one-to-one relations, incorrectly answered samples, and relations with fewer than five test samples.One-to-one mappings do not permit a test split with unseen subject/object pairs; few-sample relations are excluded to support robust evaluation.Source: §4 · p. 5

4.2 Classification accuracy and causality

• LRC performs best on both classification accuracy and causality, with non-terminal object-layer use improving accuracy over the final layer.Table 5 reports results for Llama2-7b and GPT-J; the prose highlights the improvement from object layer 21 rather than Llama2-7b’s final layer 31.Source: Table 5 · p. 6, §4.2 · p. 6
• LRC classification accuracy exceeds SVM on most, but not all, Llama2-7b relations.Figure 3 presents the relation-by-relation accuracy comparison.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3 Multi-token vs single-token objects

• Using earlier object layers improves performance for both single-token and multi-token objects, with a larger difference for multi-token objects.Figures 4 and 5 show the object-layer comparisons for multi-token and single-token objects, respectively.Source: §4.3 · pp. 6–7, Figure 4 · p. 7, Figure 5 · p. 7

4.4 Impact of the rank of the LRE inverse

• A low-rank LRE inverse improves performance dramatically, with performance peaking around rank 200 for Llama2-7b.At rank 200, the inverse discards over 95% of the 4096-dimensional weight matrix; the authors suggest important relational components are captured in its largest singular values.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5 Choosing samples to train the LRE

• Training the LRE on a sample containing the LRC’s same object results in dramatically worse performance than using a different object from the same relation.The authors suspect different-object samples may regularize the LRC, but say the phenomenon needs further investigation.Source: §4.5 · p. 7, Table 6 · p. 7

4.6 Causality vs accuracy trade-off

• Causality is highest with earlier subject layers, while classification accuracy increases up to layer 19.The authors suggest combining LRCs learned at different layers may improve both measures.Source: §4.6 · p. 8, Figure 7 · p. 8

• Individual neurons can respond to multiple concepts, complicating clean interpretation, while transformers may represent more concepts than they have hidden-layer neurons [1, 2, 3].Source: §5 · p. 8
• ROME edits facts by updating mid-layer MLPs, whereas REMEDI edits outputs by adding a vector to a subject during inference [4, 5].REMEDI’s goal is output editing, unlike this paper’s goal of identifying concept directions and building a classifier.Source: §5 · p. 8
• Probing classifiers operate on hidden activations; prior work covers vision with TCAV and textual games with Li et al. (2021), but not arbitrary human-interpretable concepts beyond those games [6, 7, 8].Source: §5 · p. 8
• The closest predecessor, Hernandez et al. (2023b), maps a subject-token activation to the object’s first output token, limiting its handling of multi-token objects and targeting relation estimation rather than concept classification.Objects sharing a first token, such as “Bill Gates” and “Bill Clinton,” are indistinguishable under the cited faithfulness evaluation.Source: §5 · p. 8

• LRCs outperform standard linear classifiers such as SVMs on both classification accuracy and causality.Source: §6 · p. 8
• Performance varies with the training samples chosen, and the authors expect optimizing LRE training samples to improve LRCs.Source: §6 · p. 8
• Causality is highest with earlier layers, while classification accuracy rises in the opposite direction, creating a layer-dependent trade-off.Source: §6 · p. 9
• The authors suggest concept-identification techniques such as LRC may support future study of relations between concepts in model weights and extraction of knowledge or world models.Source: §6 · p. 9

Limitations

• Each (r, o) pair requires its own LRC, so the method cannot generalize to new objects without a training sample for that pair.The evaluation also assumes each subject maps to only one object per relation; handling multiple objects, such as a movie’s genres, would require modifications not investigated here.Source: Limitations · p. 9
• The method assumes concepts correspond to hidden-space directions and that correct model answers reflect represented concepts, but models may instead guess correctly.For example, GPT-J may answer “Clint Eastwood” for almost any made-up person named Eastwood; distinguishing guessing from knowing is challenging.Source: Limitations · p. 9
• The method can fail when subject-object knowledge resides in attention values and is added to output tokens rather than the subject-token residual stream.It assumes the relevant knowledge can be found in the subject token’s residual stream.Source: Limitations · p. 9
• LRCs handle only relational concepts of the form (s, r, o); other concept types require an adaptation or a different technique.Source: Limitations · p. 9

Ethics statement

• LRCs may help locate biases and inaccurate information inside model weights, but do not provide a way to robustly correct them.Source: Ethics statement · p. 9

A.1 Extended results

• LRC outperforms SVM on most, but not all, tested relations in GPT-J classification accuracy.Figure 8 breaks results down by relation.Source: Figure 8 · p. 10, §A.1 · p. 10
• Figure 9 examines GPT-J classification accuracy and causality across LRE inversion ranks; its shaded area indicates standard deviation across five seeds.Source: §A.1 · p. 10, Figure 9 · p. 11
• Figure 10 examines GPT-J classification accuracy and causality across LRE object layers with subject layer 15; multi-token-object results are not broken out because GPT-J answers very few such prompts correctly.The figure reports standard deviations across five seeds.Source: §A.1 · pp. 10–11, Figure 10 · p. 11
• Figure 11 compares GPT-J classification accuracy and causality across LRE subject layers, with object layer 20; earlier subject layers improve causality at the expense of classification accuracy.The figure reports standard deviations across five seeds.Source: §A.1 · p. 11

A.2 Statistical significance

• Significance uses two-proportion Z-tests on each of five random train/test splits separately, avoiding double-counting samples across splits.This is described as more conservative than summing results across splits.Source: §A.2 · p. 11
• Significance scores are not reweighted by relation, so relations with more samples receive no additional weight and per-iteration scores differ slightly from the paper’s main results.The appendix shows GPT-J subject-layer results in Figure 11.Source: §A.2 · p. 11, Figure 11 · p. 11
• LRC is statistically significantly better than SVM on both classification accuracy and causality for Llama2-7b, with all reported p-values below 0.05.Table 7 reports comparisons at subject layer 17, using object layer 21 for LRC.Source: §A.2 · p. 11, Table 7 · p. 12
• For GPT-J, LRC’s classification-accuracy difference from SVM is not statistically significant, while its causality advantage is significant.Table 8 reports comparisons at subject layer 14, using object layer 20 for LRC.Source: §A.2 · p. 11, Table 8 · p. 12