Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers

1 Introduction

• The motivating question is whether words such as English “cat” and French “chat” map to a shared feline concept representation or separate language-specific representations.Prior work is described as hinting at language-agnostic concept representations.Source: §1 · pp. 1–2
• The study uses activation patching to test representations by inserting activations from one forward pass into another and observing the output.The paper frames this as a causal alternative to observational methods such as embedding similarity, probing, and the logit lens.Source: §1 · p. 2
• Activation patching of Llama 2 7B indicates that translation resolves the output language before the concept to be translated.The paper also tests whether language and concept information can be manipulated independently, and whether concepts have shared or language-specific representations.Source: §1 · p. 2
• Patching the mean representation of a concept across languages improves Llama 2 7B’s word-translation performance, supporting the independent-language-and-concept hypothesis.The authors report that the observations generalize across models varying in size, architecture, and training data, and that a model can generate a definition of a mean representation.Source: §1 · p. 2
• The paper presents causal evidence that LLMs actively use language-agnostic concept representations during text generation, within the investigated models and tasks.It also presents activation patching as a framework for causal investigations beyond embedding similarity, probing, and logit lens approaches.Source: §1 · p. 2

2 Related Work

• Prior work reports language-agnostic representations in encoder-only models and cross-lingual generalization in decoder-only models, alongside both language-agnostic and language-specific neurons in LLMs.The cited approaches include embedding similarity, probing, neuron analysis, and early decoding with the logit lens.Source: §2 · p. 3
• Embedding similarity, probing, and logit-lens analyses are not causal, while neuron-level analyses study causal effects of neurons rather than representations themselves.The authors position their work as addressing this gap.Source: §2 · p. 3
• The paper repurposes the LLM itself to translate or define its representations using activation patching, rather than training it to generate descriptions of them.This approach builds on definition modeling and recent patching setups that use a model to analyze its own internals.Source: §2 · p. 3
• In multilingual factual-recall tasks, Fierro et al. (2025) report the opposite order of concept and language entry from this paper’s translation task.They also find that large parts of factual recall are handled language-agnostically, despite studying models trained on a more balanced language split.Source: §2 · p. 3

3 Background

• A decoder-only transformer embeds each input token as a d-dimensional vector that begins the residual stream, which is updated layer by layer at each token position.The layer operation is described as typically comprising self-attention followed by a feedforward network.Source: §3 · p. 3
• Activation patching copies activations from a source forward pass into a target forward pass, and output-distribution changes provide evidence about information in the patched activation.The intervention can be applied at specified positions and layers.Source: §3 · pp. 3–4
• The translation prompts are few-shot tasks that map a concept from an input language to an output language, ending with the model translating the target concept.The paper denotes the prompt as TP(ℓ(in), ℓ(out), C) and also defines a concept prompt cut at the last token of the source-language concept.Source: §3 · p. 4
• The authors expect the last token of the source-language concept in the prompt to contain its latent representation.They distinguish the abstract concept from its language-specific versions and define word sets for expressions of a concept in each language.Source: §3 · p. 4
• Translation performance is tracked through the next-token probability assigned to starting tokens of the concept’s possible words in the output language.The paper uses BabelNet expressions rather than only GPT-4 translations to capture multiple possible translations when computing this probability.Source: §3 · p. 4

4 Exploratory analysis with patching

• The analysis asks whether a concept representation can be separated from its language, allowing either component to vary without changing the other.The proposed decomposition assigns concept information to a subspace and language information to its orthogonal complement.Source: §4 · pp. 4–5

4.1 Experimental design

• The experiment patches the last-token residual stream from one five-shot translation prompt into the corresponding position of another, one layer at a time.The paired prompts use different concepts and input and output languages.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Results

• Patching decodes the target concept in the target language at layers 0–11, then the target concept in the source language at 12–16, and the source concept in the source language at 16–31.Source: §4.2 · p. 5
• The layer pattern remains compatible with two accounts: concepts may be represented independently of language, or may remain entangled with language during translation.Both accounts compute the output language first, but differ in whether the model then identifies a language-independent concept or translates a language-specific concept.Source: §4.2 · p. 6

Results

• The experiments extend the earlier analysis to test both the H1-or-H2 distinction and whether the findings generalize to other models.The section also varies source input and output languages in its experiments.Source: §5 · pp. 6–7

Further evidence experiment

• The test averages representations of the same concept from multiple input languages; H1 predicts the concept survives averaging, whereas H2 predicts an incoherent mixture and translation interference.The authors patch the averaged activations into the target prompt at the concept token and subsequent layers.Source: §5 · p. 6
• Patching source-concept latents at the concept token successfully decodes the source concept in the target language, supporting either H1 or H2.The reported patch spans layers 0–15, until the last-token latents stop attending to the concept token.Source: §5 · p. 7

Results

• Patching the mean concept representation across source languages preserves and increases the probability of predicting the source concept in the target language.The authors interpret this as a concept-denoising effect consistent with H1 rather than the interference predicted by H2.Source: §5 · p. 7

Other models

• The same improvement from averaging concept representations across languages appears in Mistral 7B, Llama 3 8B, Qwen 1.5 7B, Llama 2 70B, and Aya 23 8B.The authors interpret this pattern as evidence that these models use language-agnostic concept representations.Source: §5 · p. 7

6 Generating descriptions for latents

• The experiment evaluates whether models can describe mean multilingual concept representations in natural language, using definition generation as an evaluation of semantic representations.The approach builds on definition modeling, which uses natural-language generation to evaluate what semantic information embeddings capture.Source: §6 · p. 7

6.1 Definition prompt

• The definition prompt DP(ℓ, C) is a few-shot prompt asking the model to describe concept C in language ℓ.The dataset uses BabelNet definitions for the first associated concept for each of 200 picturable words from the Basic English word list.Source: §6.1 · pp. 7–8

6.2 Patching setup

• For each target concept, the setup averages last-token latents from source prompts and patches them into the target prompt's concept token before generating a definition.Source prompts can come from translations or definitions; the patch is applied across layers.Source: §6.2 · p. 8, Figure 5 · p. 8

6.3 Experiment

• Definition quality is assessed by semantic similarity to the mean embedding of ground-truth definitions, alongside comparisons with prompting, ground-truth, and repeat-word baselines.The evaluation also compares similarity with definitions of other concepts in the dataset.Source: §6.3 · p. 8, §6.4 · p. 8

6.4 Results

• Patching concept representations across languages produces high-quality definitions comparable to or better than direct prompting, while mean representations perform slightly better.Comparable results from definition and translation prompts suggest similar concept representations across those prompt types; the authors also report the same trends in other models and languages.Source: §6.4 · pp. 8–9

7 Conclusion

• In word-level translation prompts, transformers use language-agnostic latent representations of concepts, and translation performance improves when concepts are represented by the mean across multiple languages.The authors argue that this outcome is consistent with shared rather than language-specific concept representations.Source: §7 · p. 9
• The authors suggest that shared concept spaces may support more efficient cross-lingual transfer and multilingual training, while biases may propagate through that shared space.They frame these as directions for future research and implications of their results.Source: §7 · p. 9

• The experiments cover simple concepts, leaving language-specific concepts such as “Waldeinsamkeit” unexamined.The authors describe it as a feeling of solitude and connectedness to nature when alone in the woods.Source: Limitations · p. 9
• More fine-grained probing is needed to determine how much concepts specialize by language and whether subtler language–concept entanglement occurs.Source: Limitations · pp. 9–10

A Patchscope experiment

• The patchscope lens experiment provides additional evidence about the layer from which the source concept can be decoded.The authors report that its results corroborate Section 4.Source: A Patchscope experiment · p. 12

B Translation Pair Construction

• Translation pairs use different concepts and exclude token collisions across paired concepts and languages.Pairs are shuffled before selection to reduce systematic biases in concept or language selection.Source: B Translation Pair Construction · p. 13
• The experiments typically use 200 pairs drawn from 200 picturable words in the Basic English word list.The passage connects this choice to statistical power and computational feasibility.Source: B Translation Pair Construction · pp. 13–14

C Random prompt task experiment

• The random-source experiments progressively move away from translation prompts while retaining a target translation prompt.The patching setup otherwise resembles Section 4.Source: C Random prompt task experiment · p. 14
• With randomized concepts and languages but the same template, patching layers 0–11 causes no observed accuracy drop.The authors take this as evidence that the last-token latent at those layers contains no translation-task-specific information.Source: C Random prompt task experiment · p. 14
• Using an empty-context source prompt causes target-concept probability to drop starting at layer 4.The authors attribute this to the task-specific information in the quotation mark being replaced by an empty-context token.Source: C Random prompt task experiment · p. 14
• Replacing quotation marks with @ lowers performance in early layers, while patching layers 5–11 has little effect.The authors postulate that position-marker tokens have already mapped to a general feature shared across the source and target passes.Source: C Random prompt task experiment · pp. 14–15
• Randomly shuffling the modified source prompts makes target-concept probability very low, though not zero.The authors interpret this as showing that the task cannot be solved without the position-marker feature.Source: C Random prompt task experiment · pp. 14–15
• The broader appendix reports experiments on Mistral 7B, Llama 3 8B, Qwen 1.5 7B, and Llama 2 70B.Aya 23 8B is also included for the mean-patching experiment.Source: D Other models and languages · p. 15

D.1 Exploratory analysis

• Llama 3, Mistral 7B v0.3, and Qwen 1.5 7B show a smaller target-language spike than Llama 2 7B, hinting at more overlap in output-language and concept computation.Source: §D.1 · p. 15

D.2 Ruling out hypotheses

• A mean over five few-shot contexts from one language pair does not increase target-concept probability, while a mean over language pairs does.The authors suggest language-pair averaging may increase the signal about the source concept; additional contexts from the same pair do not add that information.Source: §D.2 · p. 15

D.3 Similarity comparison

• Across additional languages and models, Figure 13 shows the same embedding-similarity trends as Figure 6.Source: §D.3 · p. 15, Figure 13 · p. 20
• Perplexity on ground-truth definitions in Figure 14 was excluded because it seemed less granular and more sensitive to syntax than semantics.Source: §D.3 · p. 15, Figure 14 · p. 20
• Figure 13 compares definition embeddings with mean ground-truth-definition embeddings and embeddings of other concepts across Aya and Gemma language settings.Source: Figure 13 · p. 20