Source-linked AI summary
How do Language Models Bind Entities in Context?
Jiahai Feng, Jacob Steinhardt
TL;DR
The paper asks how language models represent entity–attribute bindings in context, a foundational component of compositional reasoning. It uses causal interventions to examine an abstract binding-ID mechanism and finds that IDs are vector-valued and widespread across sufficiently large studied models.
Problem
The paper asks how language models represent in-context entity–attribute associations so they can later be recalled, a problem underlying compositional reasoning.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Binding IDs provide an interpretable account of how studied language models represent symbolic knowledge in context.
Source: §7 · p. 9A position-dependent bias raises log probabilities for later-position attributes regardless of binding, complicating interpretation of position-independence tests.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
To correctly use in-context information, language models (LMs) must bind entities to their attributes. For example, given a context describing a "green square" and a "blue circle", LMs must bind the shapes to their respective colors. We analyze LM representations and identify the binding ID mechanism: a general mechanism for solving the binding problem, which we observe in every sufficiently large model from the Pythia and LLaMA families. Using causal interventions, we show that LMs' internal activations represent binding information by attaching binding ID vectors to corresponding entities and attributes. We further show that binding ID vectors form a continuous subspace, in which distances between binding ID vectors reflect their discernability. Overall, our results uncover interpretable strategies in LMs for representing symbolic knowledge in-context, providing a step towards understanding general in-context reasoning in large-scale LMs.
1 INTRODUCTION
The paper studies how language models represent entity–attribute bindings in context, a foundational skill for compositional reasoning. It identifies abstract binding IDs as a mechanism for representing those associations.
1 INTRODUCTION
- The binding problem concerns representing in-context associations, such as binding Alice to Paris and Bob to Bangkok, so they can later be recalled.Because these associations are bound in context, the paper locates binding in activations rather than weights used for factual recall.Source: §1 · p. 1
- Language models use abstract binding IDs to mark variables in the same predicate apart, assigning entities and attributes in a pair the same ID.The proposed mechanism stores these bindings in activations, and queries retrieve the attribute sharing the queried entity’s ID.Source: §1 · p. 1, Figure 1 · p. 2
- Binding IDs are vectors attached to variables by simple addition, and linear combinations of binding IDs remain valid IDs unlike random vectors.The paper describes the binding IDs as occupying a subspace in activation space.Source: §1 · p. 1
- The binding-ID mechanism appears in every sufficiently large studied Pythia and LLaMA model, increases in fidelity with scale, and transfers across tasks.The paper also reports an exception: a question-answering task uses an alternate direct-binding mechanism.Source: §1 · p. 2
2 PRELIMINARIES
The preliminaries define the entity–attribute task and the activation-level causal mediation approach used to study how language models represent context. They explain how interventions and model answers support analysis of activation roles.
2 PRELIMINARIES
- A binding-task context pairs n entities with n attributes, and querying an entity should elicit its paired attribute.The main CAPITALS task uses single-token names and countries; evaluation uses median-calibrated accuracy over 100 sampled contexts.Source: §2 · p. 2
- For each token position, Zp denotes its stacked internal activations across layers, while Zcontext collects activations across the context.The paper identifies these activations as pre-transformer layer activations, also called the residual stream.Source: §2 · p. 2
- Causal mediation analysis tests an intermediate activation’s role by replacing it with an activation from a separate context and measuring changes in model behavior.The intervention modifies the target context’s activations while keeping its other components fixed.Source: §2 · p. 3
- The analysis treats the model’s answers as evidence about which context it follows, using “believes” as shorthand rather than a claim about human-like beliefs.The paper calls the model confused when its behavior is inconsistent with any context.Source: §2 · p. 3
4. Evaluate the model’s beliefs based on the new Zintervened
The section interprets changes in model answers after an activation intervention as evidence about that activation’s role. It also notes that the analysis freezes other activations and uses “belief” only as shorthand.
4. Evaluate the model’s beliefs based on the new Zintervened
- If replacing ZA0 with source-context Z′A0 shifts the model toward the source context’s answers, the intervention indicates that ZA0 carries context information relevant to behavior.The paper frames retaining the original context and fully adopting the source context as hypothetical extremes.Source: §4 · p. 3
- “Believes” is only notation for answering in accordance with a context, not an assertion that language models have human-like beliefs.Source: §4 · p. 3
- The intervention freezes other activations, such as ZE1 and ZA1, rather than recomputing them, so it suppresses their downstream causal influence.This isolates the activation substitution from changes that would otherwise propagate through the context representation.Source: §4 · p. 4
3 EXISTENCE OF BINDING IDS
The proposed binding ID mechanism assigns a shared, value-independent identifier to each entity–attribute pair, then retrieves the attribute matching an entity’s identifier. Experiments test whether activations support this account through factorizability and position independence.
3 EXISTENCE OF BINDING IDS
- The mechanism assigns each pair a shared binding ID, encodes it with both the entity and attribute, and retrieves the attribute with the matching ID.The proposed IDs are independent of entity and attribute values.Source: §3 · p. 4
3.1 FACTORIZABILITY OF ACTIVATIONS
- After swapping A0 for A′0, querying E0 favors A′0 while querying E1 still favors A1, supporting factorizability.When both entity and attribute are swapped, querying E′0 returns A′0, while querying E0 yields approximately uniform predictions.Source: Figure 3 · p. 5, §3.1 · p. 5
3.2 POSITION INDEPENDENCE
- Swapping entity or attribute positions produces little change in mean log probabilities, as position independence predicts.The authors report almost no change for entities and only a small change for attributes.Source: Figure 4 · p. 5, §3.2 · p. 6
- Attributes shifted to later positions have higher log probabilities regardless of binding, which the authors interpret as position-dependent bias rather than evidence against position independence.They describe this as an overall gradual trend in the attribute results.Source: §3.2 · p. 6
4 STRUCTURE OF BINDING ID
The section tests whether binding IDs are additive vectors and whether their geometry forms a continuous, discernible space. Interventions support additivity, while linear combinations can remain valid binding IDs and separation predicts how well the model distinguishes them.
4.1 ADDITIVITY OF BINDING FUNCTIONS
- The additive hypothesis represents each entity or attribute as its content vector plus a binding ID vector.The section extracts difference vectors between binding IDs to test the hypothesis through interventions.Source: §4.1 · p. 6
- Mean interventions support additive binding functions: control accuracy exceeds 99%, attribute and entity interventions fall below 3%, and applying both restores accuracy above 97%.Random vectors of the same magnitude have no effect on model behavior.Source: §4.1 · p. 7, Table 1 · p. 7
4.2 THE GEOMETRY OF BINDING ID VECTORS
- The geometry experiment erases binding information, then adds candidate vector pairs; restored accuracy marks a candidate pair as a valid binding ID.Candidate vectors are generated as linear combinations of two binding-ID difference-vector pairs.Source: §4.2 · p. 7
- Linear combinations of binding IDs, including negative-coefficient combinations, can restore accuracy; nearby IDs yield near-chance accuracy, while sufficiently separated IDs yield high accuracy.The results support a continuous binding subspace with a metric structure: nearby vectors are hard to distinguish, but farther-apart vectors are reliably distinguished.Source: §4.2 · p. 7
- The section speculates that attention heads may compare binding ID vectors, with attention scores potentially providing a metric over the binding subspace.This is proposed as a possible circuit-level explanation, not established by the experiments.Source: §4.2 · p. 8
5 GENERALITY AND LIMITATIONS OF BINDING ID
The section evaluates binding IDs across models and tasks, then tests whether vectors estimated on one task transfer to another. Interventions behave as expected on most tasks, and source-task vectors often restore performance on CAPITALS.
Generality of binding ID vectors
- Mean interventions induce the expected behavior on most tested tasks, and their effectiveness increases with model scale.The evaluation uses median-calibrated accuracy on three-entity contexts under control, entity-intervention, and attribute-intervention conditions.Source: §5 · p. 8
Transfer across tasks
- Binding vectors estimated from other tasks consistently yield high accuracy on CAPITALS, while zero-vector and random-vector baselines produce chance accuracy.The tested source tasks include SHAPES and PARALLEL, which have limited surface similarity to CAPITALS.Source: §5 · p. 8
- Cross-task transfer, together with earlier results, suggests that model activations may contain a task-agnostic subspace for binding vectors.The passage frames this as a possibility suggested by the results.Source: §5 · p. 8
6 RELATED WORK
The related-work discussion distinguishes this study of context-learned relations in activations from work on pretrained factual recall in weights. It also contrasts LM-constructed binding with direct binding and places the work alongside broader studies of symbolic representations and mechanistic interpretability.
Knowledge recall
- Unlike factual-recall research on pretrained relations stored in weights, this work studies context-learned relations represented and recalled through activations.The passage describes factual-recall work as spiritually related because both involve associating entities with facts.Source: §6 · p. 9
- Unlike Hernandez et al. (2023a), who construct bound representations by directly binding attribute representations to entity representations, this work studies representations constructed by the LM itself.The paper identifies binding IDs, rather than direct binding, as the mechanism LM representations predominantly use.Source: §6 · p. 9
Symbolic representations in connectionist systems
- The paper shares with symbolic-representation research the aim of uncovering how neural networks implement structured representations used in reasoning.Source: §6 · p. 9
Mechanistic Interpretability
- The work studies causal influences of representation structures on model behavior at a coarser level than circuit-focused mechanistic interpretability.Concurrent work by Prakash et al. (2024) analyzes circuits involved in the binding problem.Source: §6 · p. 9
7 CONCLUSION
The paper identifies binding as a primitive reasoning skill with a structured representation in language models. Binding IDs appear in sufficiently large studied models, and the paper argues that larger models can have more structured, interpretable representations.
7 CONCLUSION
- Binding IDs let pretrained LMs bind entities and attributes, and occupy a vector subspace whose distances reflect discernability.The paper presents binding as a skill used across several settings.Source: §7 · p. 9
- The authors identify binding as one primitive skill supporting reasoning and propose finding other primitive skills with interpretable mechanisms as future work.Source: §7 · p. 9
- Only larger models exhibited binding IDs, suggesting that representations can become more structured and interpretable as LMs grow.The authors frame this as a counterexample to the intuition that larger models are necessarily harder to interpret.Source: §7 · p. 9
A EVALUATION DETAILS
- Evaluation sampled 100 contexts and summarized performance separately for each query entity.Source: A EVALUATION DETAILS · p. 12
B NECESSITY OF BINDING ID MECHANISM
The section argues that factorizability and position independence imply a binding ID mechanism: entities and attributes can be represented with binding functions keyed by IDs, and queries recover their pairings from those IDs. It also describes position-bias adjustments and task variants used to test the mechanism’s scope.
B NECESSITY OF BINDING ID MECHANISM
- Factorizability and position independence jointly imply binding functions that represent entities and attributes by their binding IDs and support queries that recover their pairings.The section gives an informal argument for this implication.Source: B NECESSITY OF BINDING ID MECHANISM · pp. 12–13
C DETAILS FOR POSITION INDEPENDENCE
- Position independence is tested by changing apparent activation positions with RoPE, whose position information is applied at every layer rather than injected into the residual stream.Source: C DETAILS FOR POSITION INDEPENDENCE · p. 13
- The authors mitigate position-dependent bias by median calibration or by setting all attribute activations to the same position; neither procedure requires ground-truth predicates.Source: C DETAILS FOR POSITION INDEPENDENCE · p. 13
D.1 CAPITALS
- CAPITALS pairs names with countries whose capitals are queried, so success requires both retrieving the capital and binding it to the queried entity.Source: §D.1 · p. 14
D.2 PARALLEL
- PARALLEL places two entities together with two attributes, breaking the CAPITALS format’s same-sentence pairing and testing whether binding IDs are merely syntactic.Source: §D.2 · p. 14
D.3 FRUITS; D.4 SHAPES
- FRUITS changes the attributes to common foods, while SHAPES reverses the usual noun-entity and adjective-attribute roles by pairing colors with shapes.Source: §D.3 · p. 14, §D.4 · pp. 14–15
D.5 BIOS
- BIOS uses sentence-length biographical descriptions as attributes and asks the model to infer occupations, a harder task where mean attribute interventions are not expected to work.Source: §D.5 · p. 15
E MCQ TASK
- The MCQ setup places sentiment options before a review so labels must be bound to option texts, and the authors identify direct binding as an alternative to binding IDs.In this mechanism, the model directly binds each label to its option text.Source: E MCQ TASK · p. 15
E.1 EXPERIMENTS
In the MCQ experiments, a suffix-token intervention can change the model’s label-option belief, indicating direct binding rather than binding IDs. The section also describes cyclic-shift interventions used to test whether beliefs track shifted entity and attribute binding IDs.
E.1 EXPERIMENTS
- The reversed MCQ format is too out of distribution for LLaMA-30b to solve, while instruction-finetuned tulu-13b can solve it.Source: §E.1 · p. 16
- Substituting longer suffixes ranges from a no-intervention control with near-zero accuracy to near-perfect accuracy when the entire context is substituted.Source: §E.1 · p. 16, Figure 8 · p. 17
- Copying the “ative” and following whitespace activations nearly completely changes the MCQ belief, although those surface tokens are identical across contexts.The authors infer that these activations carry both label and option-text binding information, unlike the binding-ID account’s prediction.Source: §E.1 · p. 16
F GENERALITY DETAILS
- For cyclic shifts, entity interventions are expected to shift entity-attribute beliefs in the same direction, while attribute interventions predict the inverse shift.The reported intervention averages results over both cyclic shifts on three elements.Source: F GENERALITY DETAILS · p. 16
G GEOMETRY DETAILS
The geometry experiments test candidate binding-vector offsets using differences between binding IDs, because the vectors themselves are unavailable. The model regains the ability to distinguish candidates when they are sufficiently far apart, including candidates outside the basis vectors’ convex hull.
G GEOMETRY DETAILS
- The experiments can access binding-ID differences, but not the binding ID vectors themselves.Source: G GEOMETRY DETAILS · p. 17
- Candidate offsets are constructed as linear combinations of two entity and attribute difference vectors, then added to activations after an erasure step.Source: G GEOMETRY DETAILS · p. 17
- Near the fixed candidate, the model is near chance; sufficiently separated candidate IDs restore its ability to distinguish them, even outside the basis vectors’ convex hull.Source: G GEOMETRY DETAILS · pp. 17–18
H ONE HOP EXPERIMENT
ONEHOP augments capital-city binding with a task requiring two inference steps, and the reported results still hold on this harder task. The analysis also extends to three-term binding and additional task plots, with a tokenization-related caveat for BIOS attribute factorizability.
H ONE HOP EXPERIMENT
- ONEHOP requires inferring a person's city from their country, then using that city to retrieve the person's attribute.Source: H ONE HOP EXPERIMENT · p. 18
- On the harder ONEHOP task, LLaMA-65b still shows the reported factorizability, position-independence, and mean-intervention results.Source: H ONE HOP EXPERIMENT · p. 18
I THREE-TERM BINDING
- Three-term binding links names, countries, and hobbies, allowing the model to retrieve one term from the other two across six assignments.Source: I THREE-TERM BINDING · p. 19
- Mean-intervention median-calibrated accuracy exceeds CAPITALS across the six assignments, and intervened models perform near perfectly in most test conditions.Source: I THREE-TERM BINDING · p. 20, Table 4 · p. 20
J ADDITIONAL FACTORIZABILITY AND POSITION INDEPENDENCE PLOTS
- Additional plots reproduce the analyses on PARALLEL, SHAPES, FRUITS, and BIOS; BIOS shows entity but not attribute factorizability because its test swaps only the first token of multi-token attributes.Source: J ADDITIONAL FACTORIZABILITY AND… · p. 20