PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Capacity of Self-Attention Under Competition

Timur Mudarisov, Mikhail Burtsev, Radu State

arXiv:2609.37879v1cs.CLcs.LG

TL;DR

How many context tokens does a language model need to retain for prediction, and how does that depend on context and competition? The paper measures loss under top-ranked token selection and finds that required set size varies with context and aggregation, while geometric structure alone does not establish functional sufficiency.

  • Problem

    How many context tokens language models use, and what determines that number, remains an open question that self-attention can help study because it scores context tokens and combines their value vectors.

    Source: §1 · p. 1

  • Method

    The paper retains top-ranked tokens at each query, head, and layer without retraining, varies the set size, and measures loss increase against a tolerance.

    Source: §1 · p. 2

  • Results

    Longer context increases required set size while decreasing its context fraction, and renormalization can substantially reduce the required size.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Effective attention set size captures the combined effects of useful information, its ranking among competing tokens, and aggregation on predictive performance under a specified intervention.

    Source: §5 · p. 10

    The common set-size estimate does not identify a single subset of original context tokens or recover a local useful-set size, and average loss can conceal example-level changes.

    Source: §4 · p. 9

Abstract

from arXiv · show

How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the selected set size and measuring the increase in negative log-likelihood (NLL), we estimate the effective attention set size needed to stay within a chosen loss tolerance. Relatively small selected sets can keep NLL close to the full-attention baseline, although the required size varies across models. Attention-based selection substantially outperforms random selection. Selected sets exhibit geometric structure, although geometric separation alone does not establish that model loss is preserved. Extending context while evaluating the same prediction targets increases the required set size, while its fraction of context decreases over the tested range. Experiments with a fixed supporting fact show that additional background pushes its tokens down the attention ranking and reduces their attention mass. Renormalizing the retained weights can substantially reduce the required set size, showing that it also depends on how selected representations are combined. Conditional theoretical models explain how competition and attention-mass retention can produce growing set sizes without more distinct information to retrieve. These results provide a way to measure effective attention set size in language models and investigate its dependence on context, competition, and aggregation.

1 Introduction

The introduction frames self-attention as a way to study which context tokens support prediction, distinguishing the unknown useful set from the tokens selected by observable rankings. It measures functional sufficiency through loss under selection and examines how context, competition, and aggregation affect the required set size.

1 Introduction

  • Prior work found stronger geometric separation for attention-selected tokens than random sets, motivating tests of whether selection preserves model performance.The introduction proposes comparing geometric separation with loss degradation rather than treating structure alone as functional sufficiency.Source: §1 · pp. 1–2
  • The useful-token framework separates a context-dependent, unknown useful set from its imperfect attention ranking, so ranking errors can require retaining more than the useful set size.The framework guides analysis of observable selected sets without establishing their relationship to the unknown useful set.Source: §1 · p. 1
  • The study estimates effective attention set size by retaining top-ranked tokens without retraining and measuring loss against a chosen tolerance.The analysis compares attention and contribution rankings with random selection across nine checkpoints.Source: §1 · p. 2
  • With fixed prediction targets, longer context increases the required set size while decreasing its fraction of context; fixed-support experiments examine background competition.Natural context can also improve full-model predictions, so the context-extension experiment reflects both useful information and competition.Source: §1 · p. 2
  • Renormalizing retained weights can substantially reduce the required set size, and conditional models explain growth through ranking competition and attention-mass retention.The models include settings with a fixed required set or identical value vectors.Source: §1 · p. 2

2 Attention selection and effective set size

This section defines observable token rankings and an effective set-size measure for testing prediction loss under top-N selection. It also describes geometric comparisons and the limits of interpreting them as evidence of relevance.

2 Attention selection and effective set size

  • Attention ranking uses assigned probabilities, while contribution ranking also accounts for value-vector magnitudes before the output projection.Each token contribution is written as y_i = α_i v_i.Source: §2 · p. 2
  • The useful set is hypothesized to vary by sequence, layer, head, and query, and ranking inversions can force selection of more than K(C) tokens.The ranking-quality hypothesis applies to a reference distribution; experiments examine selected sets and consequences without directly testing the unobserved useful labels.Source: §2 · pp. 2–3, Figure 1 · p. 3
  • Geometric precision and recall use different radii around the selected aggregate, so their harmonic summary is not a conventional common-threshold F-score.At N = 1, Euclidean F1 equals 1 absent duplicate vectors, including under random selection; stabilized cosine distance need not share that boundary.Source: §2 · p. 4
  • At each causal query, head, and layer, Top-N retains the highest-scoring visible tokens; the same N limit is recomputed from current activations without retraining.The primary intervention leaves retained attention weights unchanged and zeros the others.Source: §2 · pp. 3–4
  • The effective attention set size is the smallest common N whose average relative NLL or answer-loss increase meets tolerance τ.The estimate depends on ranking errors, the combined values, and the sensitivity of subsequent computation.Source: §2 · p. 4
  • Renormalized Top-N rescales retained weights to sum to one, enabling comparison of required set size when the selected sum is rescaled.The comparison tests how set-size estimates depend on how selected representations are combined.Source: §2 · p. 4

3 Experiments

Across nine decoder-only checkpoints, the experiments compare geometric structure and loss preservation under token selection, then test how context, competition, and weight normalization affect the effective attention set size. Score-based selection generally preserves prediction better than random selection, but required set sizes vary and can grow with context or competition.

3 Experiments

  • The experiments evaluate nine decoder-only checkpoints, using language-modeling corpora and controlled BABILong retrieval with a fixed supporting fact and varied background.Geometric measurements cover four models; functional interventions are applied throughout the model.Source: §3 · pp. 4–5

3.1 Geometric structure and effective attention set size

  • Attention-selected sets show stronger geometric separation than random sets across four models, and stronger separation often accompanies greater NLL degradation.The larger Euclidean differences indicate a substantial magnitude-related component alongside directional structure; the correlations are positive along the plotted trajectories.Source: Figure 2 · p. 5, §3.1 · pp. 5–6, Figure 4 · p. 6
  • Attention- and contribution-selected sets produce lower NLL degradation than random sets at the same sizes, but required set sizes vary substantially across models.Contribution ranking most clearly reduces the requirement for Llama-2-7B; for most other models, the two rankings yield similar estimates.Source: Figure 3 · p. 5, §3.1 · p. 6

3.2 Dependence on context length

  • Longer contexts require larger selected sets for the same prediction targets, while the selected fraction of context decreases over the tested range.The added natural context improves full-model NLL, and the required size also grows when the allowed absolute NLL increase is held fixed.Source: §3.2 · pp. 6–7, Figure 5 · p. 7

3.3 Retrieval under competition

  • A conditional ranking model shows that a fixed useful set can require more selected tokens as competitors increase; the model assumes fixed useful-token scores and independently drawn competitor scores.The probability pL is that a competitor exceeds the weakest useful score.Source: §3.3 · pp. 7–8

3.4 Effect of attention normalization

  • Renormalizing retained attention weights can substantially reduce required set sizes, while theoretical results link growing sizes to preserving attention mass even without more distinct information.For fixed incoming activations, the local error under renormalization depends on the difference between normalized retained and discarded value means; it can also increase local error.Source: §3.4 · pp. 8–9

3.3 Retrieval under competition

  • With annotated support fixed, additional background lowers support tokens in the attention ranking and reduces their attention mass and recall among the 64 highest-weight tokens.Required set size increases substantially in several models, though Qwen’s weaker, nonmonotonic response shows that support displacement does not uniformly increase the functional requirement.Source: §3.3 · p. 7, Figure 7 · p. 8, Figure 6 · p. 8

4 Discussion

The discussion distinguishes effective attention set size from associative-memory storage capacity and identifies limits on interpreting geometry, theory, and loss-based measurements. The estimates describe predictive performance under specific interventions, not a universal count of tokens the model uses.

Discussion

  • Effective set size measures how many highly scored tokens must be retained to keep average prediction loss within a tolerance, complementing storage-capacity analyses of associative memory [Vaswani et al., 2017] [Ramsauer et al., 2021].The useful set remains unobserved, and attention weights alone do not establish relevance [Brunner et al., 2020].Source: §4 · p. 9
  • Geometry depends on value-vector magnitudes and directions relative to the selected aggregate, so geometric separation does not establish relevance or generally predict effective set size.The useful-token framework relates required size to useful-token count, ranking competition, and sensitivity to discarded contributions.Source: §4 · p. 9
  • Sparse-attention methods and studies of sinks, active or dormant heads, and geometric selection identify structure relevant to efficient inference [Martins and Astudillo, 2016, Peters et al., 2019] [Xiao et al., 2023, Guo et al., 2024, Shin et al., 2025].These experiments compute dense attention before selection and therefore do not demonstrate an inference speedup.Source: §4 · p. 9
  • Conditional theory offers possible explanations for context-dependent set growth through competition or attention-mass retention, even with fixed task-relevant information, but its assumptions are unverified for tested models.The experiments also do not establish a universal scaling law.Source: §4 · p. 9
  • A common set size is applied across operations even though selected positions vary by query, head, and layer, so the estimate identifies neither one original-context subset nor a local useful-set size.Other limits include partial relevance labels, average loss masking example-level changes, weak baselines, conditional threshold uncertainty, and heterogeneous layer/head sensitivity.Source: §4 · p. 9

5 Conclusion

The conclusion reports that attention-based selection preserves predictive performance better than random selection, while context extension raises required set size and reduces its context fraction. The formal details connect these measurements to useful-set recovery bounds and geometric comparisons, with explicit boundary conditions.

5 Conclusion

  • For fixed prediction targets, longer context increases required set size while reducing its context fraction; with fixed annotated support, added background lowers support ranking and increases required size in several models.Attention and contribution rankings produce substantially lower loss than random selection, while geometric separation alone does not determine a sufficient set size; renormalization also affects the estimate.Source: §5 · pp. 9–10

A Formal details of attention selection

  • The formal framework relates an unknown useful set to observable rankings, geometric separation, and prediction loss, using recovery bounds and local output-error identities to interpret the measurements.The geometric construction makes metrics depend on the selected aggregate and treatment of boundary points.Source: A Formal details of attention selection · p. 12, §A.2 · p. 12

A.1 Useful-set recovery from ranking inversions

  • For useful-set recovery, m ≤ I/(N − K + 1), and N ≥ K + I suffices for complete recovery when feasible.These guarantees concern recovery of the hypothesized useful set; their connection to prediction loss also depends on retained weights and subsequent computation.Source: §A.1 · p. 12
  • At N = K, precision, recall, and their harmonic mean are bounded under Hypothesis 1, while increasing N can compensate for ranking errors even when K remains fixed.Under a uniformly random strict ordering, the expected inversion count is K(|I_t| − K)/2, and the recovery-bound constants need not hold uniformly as background grows.Source: §A.1 · p. 12

A.2 Geometric construction and boundary effects

  • A positive geometric separation margin is equivalent to zero distance-based inversions and perfect precision, recall, and harmonic mean relative to the observable selected set.These distance inversions differ from score inversions against the unknown useful set.Source: §A.2 · p. 13
  • Geometric metrics need not vary monotonically with N because selection changes the aggregate, distances, and extremal radii; singleton separation also has stated exceptions.For Euclidean distance, a singleton has perfect metrics unless vectors are duplicated; stabilized cosine distance lacks that exact guarantee, and zero aggregate yields all distances equal to one.Source: §A.2 · p. 13
  • For positive α_i, unregularized cosine cancels individual scale while retaining aggregate-direction dependence; this cancellation is only approximate with the implemented regularizer.Small discarded Euclidean contributions lie near the selected aggregate’s radius, but this property alone does not imply separation; the aggregate is an unnormalized partial attention output.Source: §A.2 · p. 13

A.3 Local output error under selection and renormalization

The local analysis compares deletion and renormalization errors for fixed incoming activations, then relates ranking choices to bounds on those errors. These local guarantees motivate the rankings but do not guarantee optimal prediction loss; experimental protocols use distinct samples and aggregation rules.

A.3 Local output error under selection and renormalization

  • Contribution ranking minimizes the contribution-norm upper bound, while attention ranking maximizes retained mass and minimizes the mass-based expression.The contribution-norm bound is at least as tight for a fixed subset, but neither bound generally orders actual errors across subsets because discarded vectors can cancel.Source: §A.3 · p. 14
  • These local optima motivate the rankings but do not guarantee optimal prediction loss, which also depends on perturbation direction and downstream computation.Alignment can amplify error at fixed discarded energy, while cancellation can reduce it.Source: §A.3 · p. 14
  • Renormalization reduces local error when the discarded and retained weighted means satisfy ∥µ_T − µ_S∥_2 < ∥µ_T∥_2, and increases it under the reverse inequality.Translating all values adds (1 − M_S)b to deletion error but leaves renormalization error unchanged; when M_S = 1, both errors vanish.Source: §A.3 · p. 14

B Experimental protocols

  • The geometric, fixed-length language-model, and matched-target context-length studies use distinct samples and aggregation rules.The protocols specify separate sampling, scoring, intervention, and uncertainty procedures for each experiment.Source: B Experimental protocols · p. 15

B.1 Geometric evaluation

The appendix specifies the geometric and fixed-length language-modeling evaluations, then explains how threshold estimates are found and what they do not establish.

B.1 Geometric evaluation

  • 256 OpenWebText and 254 WikiText-103 documents per model supply 1024-position windows for final-query geometry across layers and heads.Selected-set sizes range from 1 to 256.Source: §B.1 · p. 15
  • 16 nested random rankings provide controls, with each set evaluated around its own aggregate and document-bootstrap intervals based on 2000 replicates.Aggregation proceeds from repeats to heads and layers, then documents.Source: §B.1 · p. 15

B.2 Fixed-length language modeling and threshold estimation

  • Fixed-length evaluation uses 50 documents per model and corpus, with 1024 positions and 1023 predicted target tokens per window.Interventions recompute selected sets at each query, head, and layer.Source: §B.2 · p. 15
  • Adaptive threshold search refines observed failure-to-pass crossings, but nonmonotone loss means it cannot certify the minimum over every integer set size.Cross-corpus summaries use the maximum of the two estimates and have no confidence-bound interpretation.Source: §B.2 · pp. 15–16

B.3 Matched-target context-length evaluation

Across corpora and rankings, score-selected sets separate geometrically from random controls, while effective set-size estimates vary across models. The appendices specify matched-target and controlled-QA protocols and qualify how these measurements should be interpreted.

B.3 Matched-target context-length evaluation

  • Matched-target context evaluation uses nested suffixes at four lengths, scoring the same final 128 targets within each model–corpus pair.Bootstrap intervals are conditional on evaluated selected-set sizes and do not account for unmeasured crossings.Source: §B.3 · p. 16

B.4 Controlled QA, candidate scoring, and support measurements

  • Controlled QA varies background around a fixed annotated supporting fact and measures support-token recall and attention mass alongside constrained candidate-answer loss.Validation yields 86 examples per background; scoring uses six-way candidate choice.Source: §B.4 · pp. 16–18

B.5 Implementation and numerical validation

  • The implementation retains each checkpoint’s native attention computation and output projection, with no parameter training.The Mistral-Small-24B comparison uses an 8-bit-weight setup and therefore does not isolate model-size effects.Source: §B.5 · p. 18

C.1 Geometric separation across corpora and rankings

  • Score-selected sets show stronger separation than same-size random sets across both corpora and rankings, with a larger advantage under Euclidean geometry.The Euclidean differences indicate a substantial magnitude-related component alongside direction.Source: §C.1 · p. 18, Figure 8 · p. 19, Figure 9 · p. 19

C.2 Geometry and functional degradation

  • Contribution-ranking geometry and full-model NLL degradation vary together across selected-set sizes, but use different query populations and do not establish operation-level prediction.Excluding small set sizes weakens most correlations, and excluding sizes up to four can reverse their signs.Source: §C.2 · pp. 18, 21

C.3 Effective attention set size by corpus

  • Gemma-7B requires the largest selected sets on both corpora, while Mistral-7B and Llama-3 checkpoints reach the same tolerances with substantially smaller sets.For Llama-2-7B at 5%, contribution ranking lowers estimates from 85 to 60 on OpenWebText and from 101 to 65 on WikiText-103.Source: Table 2 · p. 21, §C.3 · p. 23

C.4 Context-length dependence and scoring protocol

With matched final targets, longer contexts require larger attention-selected sets, although the retained fraction falls over the measured range. Estimates also depend on scoring protocol, and the tested lengths do not establish a universal growth law.

C.4 Context-length dependence and scoring protocol

  • An eightfold context increase raises the 5% attention-ranked set-size estimate 4.12–6.63× across six complete model–corpus pairs.The estimate uses matched final-target scoring.Source: §C.4 · p. 23, Figure 13 · p. 23, Table 3 · p. 23
  • The retained fraction N*/L decreases over the measured range, but four tested lengths do not establish a universal sublinear or asymptotic growth law.Llama-3-8B on WikiText-103 follows N*(L) = 2 + L/32 exactly at the four tested lengths.Source: §C.4 · p. 23, Figure 14 · p. 24
  • At L = 2048, all-token scoring yields lower 5% attention estimates than final-128-target scoring for Qwen, Gemma, and Llama on both corpora.The paired estimates are 78/51, 504/394, and 57/30 on OpenWebText, and 70/48, 423/332, and 66/36 on WikiText-103.Source: §C.4 · p. 24, Table 4 · p. 24, Figure 15 · p. 25
  • Contribution ranking preserves context-length growth, but its small differences from attention ranking do not establish a consistent ranking advantage.At L = 2048 on OpenWebText, its 5% estimates are 74, 503, and 56 for Qwen, Gemma, and Llama.Source: §C.4 · p. 24
  • Full-model NLL falls by 0.292–0.342 nats per token from L = 256 to 2048, while set sizes still grow under a fixed absolute tolerance.On OpenWebText, fixed-tolerance estimates rise from 16 to 72 for Qwen, 76 to 496 for Gemma, and 10 to 54 for Llama.Source: §C.4 · p. 25
  • Mistral-7B-v0.3 has a provisional OpenWebText 5% estimate of 50 at L = 2048, but its refinement is incomplete and the value is excluded from complete-model comparisons.The completed estimates at L = 256, 512, and 1024 are 9, 13, and 22.Source: §C.4 · p. 25

C.5 Mistral-Small-24B results

Mistral-Small-24B extends checkpoint coverage, with low-tolerance set-size estimates reported under attention and contribution rankings. Its 8-bit-weight configuration means the comparison does not isolate model-size effects.

C.5 Mistral-Small-24B results

  • At 1%, 5%, and 10% tolerances, the maximum across the two corpus-specific estimates is 119, 40, and 22 under attention ranking and 120, 39, and 22 under contribution ranking.The refined loss curves show local nonmonotonicity near the strict 1% boundary.Source: §C.5 · p. 26, Figure 17 · p. 26
  • The 8-bit-weight configuration means this checkpoint comparison does not isolate the effect of model size.The configuration uses mixed-precision computation.Source: §C.5 · p. 26, D Additional controlled-retrieval… · p. 26

D Additional controlled-retrieval results

  • The accompanying controlled-retrieval analysis holds annotated support fixed while varying background text and reports support statistics, answer-loss thresholds, and full-model baselines.A supplementary multi-fact comparison examines limits on interpreting thresholds as evidence about useful-set size.Source: D Additional controlled-retrieval… · p. 26

D.1 Support displacement, mass, and recall

With fixed annotated support, background growth sharply increases support displacement and reduces its attention mass and recall, while functional set-size estimates respond differently across models. Background also raises full-model answer loss, and multi-fact estimates require caution because qa3 baseline accuracy is near chance.

D.1 Support displacement, mass, and recall

  • At 4K versus 0K, mean support displacement rises 50.0×, support attention mass falls to 0.435× its initial value, and mean support recall@64 drops 63.8 percentage points.Displacement ratios range from 42.8× to 54.6× across models; log–log slopes against non-support counts are 0.79–0.84 across the four measured background conditions.Source: §D.1 · p. 27
  • As non-support tokens increase approximately 108–111× from 0K to 4K, each competitor becomes less likely to outrank support, but more competitors outrank support overall.Mean empirical outranking probability falls for all four models, from 0.475 to 0.224 for Qwen, 0.451 to 0.226 for Gemma, 0.443 to 0.172 for Llama, and 0.451 to 0.205 for Mistral.Source: §D.1 · p. 27, Figure 18 · p. 27

D.2 Extended answer-loss thresholds

  • Qwen’s 0.10-nat estimates are 32, 16, 32, and 64 across the four backgrounds, showing that support displacement need not produce a monotone functional response.At 4K, adaptive searches resolve crossings at N = 316 for Llama-3-8B and N = 760 for Gemma-7B; Gemma at 2K and Mistral at 4K remain unresolved through 256.Source: §D.2 · pp. 27–28, Table 6 · p. 28

D.3 Full-model performance across backgrounds

  • From 0K to 4K, mean candidate loss rises for all four models, while accuracy falls by 23, 18, 10, and 14 percentage points for Qwen, Gemma, Llama, and Mistral, respectively.Llama accuracy improves at 1K, and its paired 0K-to-4K accuracy-change interval includes zero, qualifying the accuracy trend.Source: §D.3 · p. 28, Figure 19 · p. 28, Table 7 · p. 28

D.4 Fact multiplicity and baseline competence

  • Full-model qa3 accuracy is 0.23–0.30 against chance accuracy of 1/6, limiting accuracy-based intervention estimates and preventing these comparisons from identifying the useful-set size.Annotated facts span several tokens and need not specify every useful token.Source: §D.4 · p. 29, Figure 20 · p. 29, Table 8 · p. 29

E.1 Renormalization of selected weights

Renormalization can sharply change effective set-size estimates, while comparisons across models, layers, and diagnostics reveal substantial heterogeneity. Positional selection often fails to meet the tested loss criterion, and conditional models frame these measurements without identifying a single mechanism.

E.1 Renormalization of selected weights

  • 226 to 7: Gemma-7B’s 5% attention-ranked estimate falls under renormalization, the largest reported change.The corpus-level comparison uses OpenWebText deletion estimates and a renormalized evaluation; historical document pairing is unverified.Source: Table 9 · p. 30, §E.1 · p. 30
  • At 0.10 nats under attention ranking, renormalized 4K/0K set-size ratios are one for Qwen and Gemma and two for Mistral; deletion ratios are two and 11.875 for Qwen and Gemma.Mistral’s deletion search is unresolved through 256 at 4K, so its plotted endpoint is not a resolved ratio.Source: §E.1 · p. 30, Table 10 · p. 31, Figure 21 · p. 31

E.2 Comparison within the Gemma family

  • Gemma-2-9B estimates are (12, 7) for attention ranking and (11, 7) for contribution ranking, versus Gemma-7B’s (226, 206) and (227, 205), respectively.The approximately 19–29-fold differences show that family identity and parameter count alone do not account for Gemma-7B’s requirement; the comparison does not isolate architectural or training changes.Source: §E.2 · p. 31

E.3 Layer- and head-localized interventions

  • At N = 128, Gemma layer 0 has a reported 70.9% NLL increase and head 12 has a 62.1% increase, but differing sample sizes and baselines prevent assigning the head a share of the layer effect.Qwen and Mistral use different reference sizes, and their largest sampled layer effects are 1.86% and 0.33%, respectively.Source: §E.3 · pp. 31–32, Figure 22 · p. 32

E.4 Exploratory prediction of effective set size

  • 0.75 versus 1.75: mean log2 set-size error is lower for both mean and P95 sensitivity statistics than for the raw-N baseline across 16 cases.The analysis uses four models and 20 documents per model; selecting the best diagnostic requires a separate evaluation.Source: §E.4 · p. 33, Figure 23 · p. 33

E.5 Positional selection at matched set sizes

  • At the 0.10-nat tolerance, Qwen has no passing positional-control size through 256, while Mistral passes at 32 only at 0K and not through 256 at 2K or 4K.The control retains the initial token and recent positions, uses deletion without renormalization, and computes dense attention before masking.Source: §E.5 · p. 34, Figure 24 · p. 34

F Conditional explanations of context-dependent set size

  • The conditional models distinguish set recovery, attention-mass retention, local-output preservation, and task-loss tolerance, but the experiments do not establish their assumptions for individual heads or an asymptotic law.The models organize the empirical findings without identifying a single mechanism.Source: F Conditional explanations of… · p. 34

F.1 Competition with a fixed required set

Under a fixed required set, ranking competition can make the number of tokens needed for complete recovery grow with context. Under a stable weight distribution, retaining a fixed attention-mass fraction also requires a context-proportional set, and additional value assumptions connect that mass to local output accuracy.

F.1 Competition with a fixed required set

  • With a fixed required set of size K and constant competitor exceedance probability p, expected complete-recovery size is K + (L − K)p and grows affinely with context length.If pL ≍ L^−γ for 0 < γ < 1, the expected excess above K is of order L^(1−γ); bounded expected size requires DpL = O(1).Source: §F.1 · p. 35

F.2 Attention-mass retention under a stable score distribution

  • Under independent positive weights with a continuous distribution and finite positive mean, the required set fraction for retaining a fixed attention-mass fraction converges to a positive limit.The Gaussian-logit example illustrates how score spread determines the retained fraction under a stable distribution; it is not fitted to empirical logits or loss-based set sizes.Source: §F.2 · pp. 35–36

F.3 From retained mass to local output accuracy

  • With uniformly bounded values sharing a positive projection onto one direction, retained-mass bounds give necessary and sufficient conditions for local output accuracy, but the upper norm bound alone does not imply a necessary set size.Connecting this local result to downstream loss requires an additional sensitivity assumption.Source: §F.3 · p. 36

F.4 Renormalization and an explicit task-loss model

Conditional examples show that deletion-based loss requirements can grow even when all values are identical, while renormalization can remove that growth. The paper also models how the scored query positions affect estimates and treats context, competition, aggregation, and evaluation criteria as joint interpretive factors.

F.4 Renormalization and an explicit task-loss model

  • With identical values and uniform weights, deletion requires ⌈cτL⌉ tokens while renormalization requires one, so growing deletion requirements need not indicate more distinct useful information.For a general downstream loss, extending the local fixed-activation comparison to full-model set size requires additional assumptions.Source: §F.4 · p. 37

F.5 Dependence on the scored query positions

  • When earlier queries are included under the assumed common degradation profile, the limiting set-size fraction is no greater than when only a final tail of queries is scored.The relation assumes unique interior crossings; relative NLL uses length- and protocol-specific baselines outside this absolute-loss comparison.Source: §F.5 · p. 37

F.6 Implications for the empirical findings

  • For fixed prediction targets, natural-context extension raises required set sizes even under a fixed absolute tolerance; in controlled backgrounds, fixed support is displaced, loses attention mass, and has lower fixed-size recall.The normalization evaluations do not provide a matched-target context-length sweep, and the latent useful-set size remains unobserved.Source: §F.6 · p. 38

Ethics Statement

  • The work uses public pretrained models and established benchmarks, with measurements limited to specified diagnostics and interventions; sparsification or cache-reduction applications should test prediction degradation on the intended distribution.Source: Ethics Statement · p. 39

Reproducibility Statement

  • Independent reproduction of empirical summaries requires raw per-document measurements, experiment code, and environment manifests.The source archive contains the manuscript and figure assets, while the mathematical results include assumptions and proofs.Source: Reproducibility Statement · p. 39

AI Use Statement

  • Language models assisted with editing, restructuring, and debugging; the authors take responsibility for the mathematical statements, experimental design, analysis, citations, and final text.Source: AI Use Statement · p. 39
Loading 2609.37879v1…