Source-linked AI summary
In-Context Binding Capacity in Language Models
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
TL;DR
The paper asks how many entity–value assignments language models can recall together, and measures that limit with continuous recall curves and a broad threshold sweep. The continuous curves show a size law, while the broader sweep shows recipe-associated differences across scale.
Problem
A model may need to remember which value belongs to each entity across simultaneous assignments, but the capacity of this in-context binding is the central question.
Source: §1 · p. 1Method
The paper measures recall as binding load varies, using continuous recall curves for some models and a threshold crossing for a broader model sweep.
Source: §2 · p. 2Results
The continuous curves show a size law for K50, while the broad sweep shows recipe-associated differences across scale and an 8× k* range of 3 to 24 after exclusions.
Source: §2 · p. 2, §11 · p. 16Takeaways & Limitations
The measurement characterizes capacity at the query interface and provides a baseline for more precise tests of state tracking, without establishing that models maintain a world model.
Source: §1 · pp. 1–2The connection to alignment is bounded: improved retrieval alone would not establish improved policy adherence, and attributing an effect to retrieval requires further mediation assumptions.
Source: §9.4 · p. 15
Abstract
from arXiv · showhide
How many assignments can a language model recall before it loses track of which value belongs to which entity? We measure this limit using continuous recall curves for 12 models at or below 3B parameters and a threshold sweep over 30 open models up to 12B. On the continuous curves, the load at which recall falls halfway to chance follows $K_{50}=cN^{\alpha}$, with $\alpha=0.820$ and $R^2=0.73$. The broader sweep shows an eightfold range associated with pretraining recipe, although the continuous curves show no detectable recipe effect after controlling for scale, with few modern models in the fit. We derive why interference can lower measured capacity by reducing single-binding recall even when the load-dependent recall profile is unchanged. Direct task training also exceeds the extrapolated zero-shot law, but different measurement criteria prevent interpreting that comparison as a capacity gain. Its formation times follow a power-law form in two independent codebases, conditional on runs that succeed. Together, these results characterize capacity at the model's query interface. Bounds on joint recall and a decomposition of policy errors connect this measurement to working memory and instruction following, without treating recall as a measure of alignment. The controlled task also provides a baseline for testing whether binding limits constrain world-state tracking; the present experiments do not measure state updates or downstream transfer.
1 Introduction
The paper measures in-context binding: whether a model recalls the value assigned to a queried entity across simultaneous assignments. It presents a zero-shot capacity law and conditions for interpreting comparisons across models and tasks.
1 Introduction
- The task isolates static retrieval of an assigned value, not updating facts or maintaining a world model as states change.The measurement can make later state-tracking tests more precise, but does not establish that a model maintains a world model.Source: §1 · p. 1
- On continuous curves at or below 3B parameters, K50 follows K50 = c N^α with α = 0.820, while the broader sweep shows recipe-associated differences across scale.The continuous fit describes a zero-shot operating regime; the two instruments cover different model ranges and have different resolution.Source: §1 · pp. 1–2
- The task varies binding load K and interference length D, with recall queried by asking for the obligation assigned to an entity.Capacity statistics are midpoint crossings between each model’s K=1 ceiling and chance; k* is measured on a grid and K50 comes from a logistic fit.Source: Figure 1 · p. 2
- Direct task training is reported as a factor of 26 relative to an extrapolation of the zero-shot law, but the different estimands prevent interpreting this as a capacity gain.The extrapolation is outside the task’s load domain.Source: §1 · p. 2
- The paper connects recall measurements to working memory and policy adherence while deriving conditions on interpretation, without treating recall as a measure of alignment.Its contributions include threshold-transfer analysis and an identity separating interference-related changes in single-binding reliability from changes in the remaining load profile.Source: §1 · p. 2
2 Measuring capacity
Capacity is measured as recall’s decline with the number of bindings, using a grid threshold for the broad sweep and a fitted midpoint for continuous curves. The broad-sweep sample excludes a model whose single-binding accuracy is near chance and reports ceiling-level values as censored.
2 Measuring capacity
- k* is the smallest binding load where recall falls below the midpoint between the model’s K=1 ceiling and chance; K50 is the analogous fitted crossing on continuous curves.K50 uses recall normalized between the model’s own ceiling and chance, with a logistic fit in log K; the estimator is validated on synthetic curves and right-censoring is reported rather than clipped.Source: §2 · p. 2
- Excluding Pythia-70M leaves n = 29 and an observed k* range of 3 to 24, or 8×, rather than the raw table’s 12× range.Its K=1 accuracy is 0.070 against chance of 0.040, so its threshold is noise; it cannot hold a single binding.Source: §2 · p. 2
- Three models reach k* = 24, the entity-pool ceiling, and are right-censored: OLMo-1-7B, Qwen2.5-3B, and OPT-6.7B.The low end of the range is measured.Source: §2 · p. 3
- The released table covers 30 open models from ten families, while continuous re-measurement covers twelve models at or below 3B across four families.All measurements are on frozen models; the interference sweep covers eight models with committed D-sweep curves.Source: §2 · p. 3
3 A zero-shot scaling law for capacity
A re-measurement of continuous recall curves finds a positive association between model size and zero-shot capacity, while the recipe effect is not detectable after controlling for scale in the smaller cohort. Alternative summaries and threshold criteria qualify how that scaling result should be interpreted.
3.1 The continuous instrument
- A re-measurement released full recall-versus-K curves for 12 models, using six seeds per cell on a K grid to 64 for models above the K=1 exclusion floor.The original and re-measured GPT-2 results differ, and models censored at K=24 in the first wave resolve to finite values on the full grid.Source: §3.1 · p. 3
3.2 The K50 power law
- K50 scales with parameter count as K50 = c N^α, with α = 0.820 (R2 = 0.73) across 11 uncensored models.The family-clustered CI is [0.674, 1.251], and the leave-one-family-out range is [0.76, 1.09].Source: §3.2 · p. 3
3.3 Recipe after controlling for scale
- At ≤3B parameters, the recipe coefficient is +0.123 (family-clustered CI [−0.521, 0.363]), spanning zero, while the scale coefficient is +0.818 (CI [0.697, 1.206]).The regression has n = 11 models across four families, including only three modern models; the threshold sweep extends to 12B, but continuous curves above 3B are unavailable.Source: §3.3 · p. 3
3.4 Sensitivity to the summary statistic
- A threshold-free AUC summary also increases with scale: its log-parameter exponent is +0.329 (R2 = 0.799) over all 12 models, with family-clustered CI [+0.286, +0.480].AUC avoids thresholding and includes the censored model, but can saturate beyond the measured range and its exponent is not comparable to the K50 exponent.Source: §3.4 · p. 4
- Eleven of 12 models have K=1 ceilings from 0.918 to 0.99; Pythia-160M is the exception at 0.589.The ceiling is only weakly related to scale (R2 = 0.259), and this summary alone does not show how ceiling uncertainty affects fitted midpoints.Source: §3.4 · p. 4
- The logistic slopes range from 0.499 to 1.801, but slope is strongly coupled to K50 (r = −0.837 in logs), so the analysis reports no separate slope scaling law.Once K50 is known, parameters add almost nothing (partial r = −0.169); fitting a finite-slope logistic does not independently rule out a sharper underlying change.Source: §3.4 · p. 4
3.5 When a scaling exponent transfers across thresholds
- For the fitted curves, a model-order reversal at a stricter recall criterion is conditional on unequal slopes and a crossing load inside both measured domains.Slope variation can also change the fitted exponent at a criterion other than the midpoint when inverse slope covaries with size; this is not an observed reversal or measured correction to α.Source: §3.5 · p. 5
4 The threshold sweep to 12B
The threshold sweep finds a recipe-associated difference in binding capacity across 29 measurable models, with substantial overlap between old and modern recipes. Model ladders and an exclusion check show that the contrast is not uniform and survives removal of the disputed family.
4 The threshold sweep to 12B
- Across 29 measurable models, k∗ spans 3 to 24; old recipes have median 3 (n = 17) and modern recipes median 16 (n = 12), with family-clustered 95% CI [+10, +13] and family-level p = 0.012.The model-level Mann–Whitney test gives p = 4.8 × 10−4 but treats siblings as independent draws.Source: §4 · p. 5
4.1 Recipe and scale
- The Pythia ladder stays between k∗= 3 and 6 across a 29× parameter range, while Qwen2.5 rises from 3 to 24 before falling to 16 at 7B.These recipe-associated ladder contrasts do not identify a causal interaction; the OLMo-2 base/SFT/DPO/Instruct ladder remains at k∗= 16 throughout.Source: §4.1 · p. 5
4.2 Overlap between recipes
- The recipe classes overlap: modern OLMoE has k∗= 6, while old OPT-1.3B reaches 10 and old OPT-6.7B reaches the censored ceiling at 24.Both classes span 3–24, so modern recipes do not uniformly dominate.Source: §4.2 · p. 6
4.3 Sensitivity to the disputed family
- Removing the three GPT-2 models shifts the old-recipe median from 3 to 4 and the median difference from +13 to +12, leaving the recipe split intact.This exploratory check reuses the existing threshold table despite the continuous instrument placing GPT-2 below the K=1 exclusion floor.Source: §4.3 · p. 6
5 The boundary: direct task training
Direct task training forms bindings beyond the extrapolated zero-shot law, but the comparison uses different estimands. Formation-time fits replicate a power-law form among successful runs, while conditioning on success does not establish an unconditional scaling law.
5.1 Direct task training
- 3.7 versus 0.14 bindings (factor 26) compares different estimands and an extrapolated denominator below the measured range, so it is not a measured training gain.The task-trained result establishes learnability in its own setting, not the zero-shot law outside its observed range.Source: §5.1 · pp. 6–7
5.2 The formation-time law
- t ≈766 K^2.58 (R^2 = 0.95) and t ≈667 K^2.12 (R^2 = 0.97) replicate a power-law form across codebases, but do not resolve a common exponent.The fits concern successful runs; the second corpus's K=8 cell has two runs, and removing it shifts its exponent to 1.7. These training-step curves do not map extra training to a change in pretrained K50.Source: Figure 4 · p. 7, §5.2 · p. 7
5.3 What conditioning on formation changes
- Completed-run medians are no greater than unconditional medians, but that ordering does not bound scaling exponents; unconditional inference also requires completion probabilities and censoring budgets.When fewer than half of runs complete within budget, the unconditional median may lie beyond the budget or be infinite.Source: §5.3 · pp. 7–8
6 Interference costs three separate things
Interference affects the recall ceiling, midpoint, and slope differently, so capacity and robustness are distinct measurements. Lower single-binding recall can reduce a fixed-reference midpoint even when load dependence relative to the lowered ceiling is unchanged.
6 Interference costs three separate things
- At K=1, the 256-token block costs a median 0.137 of the recall ceiling, ranging from 0.016 to 0.538 across 12 models; its association with capacity is inconclusive.Spearman ρ = −0.473, p = 0.142; these curves establish neither protection from greater capacity nor independence between capacity and ceiling loss.Source: §6 · p. 8
- Against the uninterfered ceiling, retained fraction has median 0.535 with reductions in 10 of 11 models; against each curve's own ceiling, it is 0.963 with reductions in 5 of 9.Own-ceiling normalization divides by a ceiling that interference has already lowered, removing that ceiling damage from the measurement.Source: §6 · p. 8
- The slope ratio at D = 256 versus D = 0 has median 1.008 across nine models, with five steeper; these data do not establish equivalent curve shapes.The observed range is 0.85 to 1.38.Source: §6 · p. 8
- At K = 8, recall falls from 0.27–0.96 at D = 0 to 0.10–0.31 at D = 256, and collapse rates do not follow capacity ranking.The D-sweep uses eight models with committed curves; the interference block is unrelated filler, unlike overwriting earlier values for the same keys.Source: §6 · p. 8, Figure 5 · p. 9
6.1 What the two normalisations identify
- If interference lowers single-binding performance while leaving relative load dependence unchanged, the own-ceiling midpoint stays fixed but the fixed-ceiling midpoint falls; this is compatible with, not proven by, the measurements.The normalized-recall identity separates retained single-binding performance above chance from the remaining load dependence.Source: §6.1 · pp. 8–9
7 What geometry can explain
The section separates model-level geometric associations with binding capacity from explanations of individual retrieval failures. It shows why recall curves and declaration-site representations do not, by themselves, identify storage or retrieval mechanisms.
7 What geometry can explain
- Declaration-site geometry is an imperfect basis for mechanistic conclusions: the measurement script was not preserved, preventing independent verification of token positions.The available measurements also report position and activation-scale artifacts; the comparisons do not identify binding storage.Source: §7 · p. 9
7.1 Recall alone does not identify storage failure
- The same recall curve can arise from different allocations of failure between retention and access, so behavior alone cannot identify which factor failed.The two-stage construction demonstrates ambiguity in interpreting recall; it does not claim that transformers literally use these stages.Source: §7.1 · p. 10
7.2 Recipe contrasts and capacity prediction are different tests
- Modern recipes have higher packing efficiency and lower pairwise interference than older recipes at the model level, but family-adjusted comparisons do not establish within-family prediction.The geometric patterns are consistent with a recipe contrast, not an identified mechanism; packing efficiency also includes capacity in its numerator.Source: §7.2 · p. 10, Table 1 · p. 10, Figure 6 · p. 11
7.3 Cross-model capacity and within-model retrieval separate
- Subspace overlap adds −0.0005 to +0.0048 AUROC beyond position across seven models; the positional term alone wins the reported head-to-head.The emitted obligation matches the maximum-overlap competitor on 0.12–0.19 of failures against a 0.20 chance rate, while the comparison includes a 256-token interference block and is not directly comparable to Gur-Arieh et al. [2026].Source: §7.3 · p. 11
7.4 An earlier entity state can still serve as an address
- An entity state preceding its assigned value cannot directly encode that later value, but it can still serve as an address associated with it by a later state.Failure to decode the obligation from that earlier state does not rule out a causal role in retrieval.Source: §7.4 · pp. 11–12
8 Using capacity comparisons
Capacity comparisons depend on the recall criterion: an absolute requirement identifies feasible load–interference pairs, while a model’s own-ceiling midpoint summarizes decline relative to its baseline. The section proposes checks that distinguish threshold, interference, geometric, and formation explanations.
8.1 Absolute recall defines the feasible loads
- A common absolute recall requirement can correspond to different retained-performance criteria for each model and interference condition, and some loads may not meet it.If the single-binding ceiling is below the requirement, no load meets it; finite-sample estimates need uncertainty for both ceilings and recall probabilities.Source: §8.1 · p. 12
- Report the feasible measured load–interference pairs with uncertainty rather than converting unmeasured loads into fitted capacity claims.This complements the midpoint, which summarizes decline relative to baseline performance.Source: §8.1 · p. 12, §8 · p. 12
8.2 Analyses that distinguish the remaining explanations
- Threshold transfer should compare crossings at fixed recall criteria on the same models, with censoring reported; reversals outside the measured range remain predictions.Direct empirical crossings can check logistic extrapolation.Source: §8.2 · p. 12
- A ceiling-only interference account predicts interfered recall as b + (CD − b)r0(K); systematic residuals across load would reject that account.Separate trials or propagated shared uncertainty are needed for ceiling estimation and profile evaluation.Source: §8.2 · p. 12
- Proposed follow-up tests use family-held-out geometric prediction, held-out-context trial prediction, and formation-time analyses that retain unsuccessful runs.These are proposed analyses, not additional results; a positive result at one explanatory level would not establish another.Source: §8.2 · p. 12
9 Working memory, interpretability, and alignment
The discussion connects binding recall to working memory and related tasks while keeping its scope distinct from world-state tracking, joint reliability, alignment, and mechanism. It identifies the additional measurements needed to test those connections without treating static recall as evidence of broader abilities.
9.1 Working memory is a task-dependent ability
- The task measures temporary access to relations supplied in context, but increasing binding count also changes token length or query distance.Matched-length comparisons and separate manipulations of retained, overwritten, or composed values are needed to isolate which operations the capacity statistic predicts.Source: §9.1 · p. 13
9.2 From binding to world-state tracking
- The trials assign one obligation per entity and ask one query, so they do not measure state updates or action outcomes.A stronger world-state test would revise properties after an event and use the revised state to predict or choose an action; matched-length and position controls are also needed.Source: §9.2 · p. 13
9.3 Single-query recall does not determine joint reliability
- Marginal recall does not determine the probability of retrieving every required binding; without independence, the same mean recall can accompany different joint reliability.A sufficient distribution-free condition for tolerated joint failure ϵ is that the sum of marginal failure probabilities is at most ϵ; the current trials do not estimate multi-action success.Source: §9.3 · p. 14
9.4 Correct retrieval and policy adherence are separate outcomes
- Retrieval reliability alone does not identify a policy-violation rate, and the entity–obligation task has no competing authority levels or policy-adherence endpoint.The accounting identity is not a causal mediation result; better retrieval reduces violations under fixed conditional rates only when e1 < e0.Source: §9.4 · pp. 14–15
9.5 Interpretability must distinguish representation from use
- A decoder tests whether a chosen representation exposes a label, not whether the model uses that information to answer.A proposed intervention would swap an entity address or later value representation and predict the selective answer change; current geometry and probes do not establish that connection.Source: §9.5 · p. 15
10 Related work
The related work distinguishes this paper’s in-context binding measure from loss scaling, working-memory tasks, long-context benchmarks, and studies of changing world states. It places transfer from binding capacity to downstream performance as an open empirical question.
Related scaling and mechanism accounts
- The K50 law describes a specific in-context capacity, unlike scaling laws that fit language-model loss against scale or data-constrained training.In-context binding state is framed as complementary to parametric knowledge in weights, and mechanism work includes induction heads and associative memories.Source: §10 · p. 15
Cognitive and long-context comparisons
- Working-memory studies measure verbal or spatial tasks, while long-context benchmarks vary retrieval conditions; this task varies binding count and distractor length but does not fully separate binding count from token length.The paper estimates a load-capacity scaling law across model sizes rather than a single shared working-memory capacity.Source: §10 · p. 15
World-state and transfer comparisons
- World-state studies test changing-state representations, whereas this static task does not; whether its capacity measure predicts downstream performance beyond model size remains untested.Synthetic retrieval training has been reported to transfer to multi-document question answering, while needle-retrieval scores do not reliably predict downstream long-context performance.Source: §10 · p. 15
11 Discussion and conclusion
Continuous curves at or below 3B show a size law without a detectable recipe effect after scale control, while the broader sweep to 12B shows recipe-associated differences. Training, interference, and measurement criteria qualify what these capacity comparisons mean.
11 Discussion and conclusion
- The recipe split is observational across 29 models that differ in data, tokenizer, optimizer, architecture, and era, and k* is right-censored at 24.The comparison identifies a recipe-level regularity without causal attribution; the low end of the range is more informative than the ceiling.Source: §11 · p. 16
- The laws apply to this synthetic single-token task, its model ranges, and its estimators; geometric correlates do not identify a trial-level mechanism.The proposed tests in Section 8 remain proposals.Source: §11 · p. 16
- Across models to 12B, k* shows recipe-associated differences, while continuous curves at or below 3B show a size law and no detectable recipe effect after scale control.The continuous-curve K50 scaling fit is post hoc and awaits pre-registered replication.Source: §11 · p. 16
- Direct task training exceeds an extrapolation of the zero-shot law by 26×, but different measurement criteria prevent interpreting this as a capacity gain.The comparison uses different estimands and a prediction outside the task’s load domain.Source: §11 · p. 16
A Measurement diagnostics for declaration-site geometry
Declaration-site geometry measures the dimensionality and overlap of entity codes, not the binding itself. Position and massive activations can distort its measurement, while the declaration site precedes the assigned obligation.
A Measurement diagnostics for declaration-site geometry
- Declaration-site ddecl measures the effective dimension of entity codes, and interference measures how much their directions overlap; neither measures binding itself.The bound obligation decodes at 0.017–0.054 against chance 0.040 in the reported models.Source: A Measurement diagnostics for… · p. 18, §A.1 · p. 18
A.1 Measurement artifacts at the declaration site
- Raw declaration-site vectors are overwhelmingly positional on 7 of 8 models, yielding effective dimension at most 1.03 and pairwise squared cosine 0.84–1.00.Removing the per-slot mean gives interference of 0.006–0.009 on all 8 models.Source: §A.1 · p. 18
- For Qwen2.5-7B, covariance participation ratio reports dimension 1.3, while per-coordinate standardization gives 35.3.One coordinate carries 54% of residual variance; standardized dimensions are 12.0, 35.3, and 81.1 for 1.5B, 7B, and 14B.Source: §A.1 · p. 18
- Across Pythia-1.4B checkpoints, positional variance rises from 0.15 to 0.99 and the top coordinate’s share from 0.001 to 0.17.The corrected estimator shows the declaration-site subspace expanding and then plateauing, while the covariance ratio suggests compression.Source: §A.1 · p. 18
- The declaration-site vector precedes its obligation, so ddecl captures entity-code dimension and packing efficiency captures how directions are spent keeping entities distinct.The entity decodes at 1.000 and the obligation at chance in the reported probes.Source: §A.1 · p. 18
- The ddecl measurement script was not preserved, so its token position cannot be independently verified; at the query site, retrieved obligations decode at 0.51–0.86.The query-site result is against chance 0.040 on 5 models.Source: §A.1 · p. 18