Content-Based Addressing for Long Context

1 Introduction

• At evaluation lengths beyond training, global positions create a positional mismatch, and a unit’s serialized position makes its rotation depend on unrelated tokens.Source: §1 · p. 1
• Each token keeps a bounded position within its unit, while each completed unit receives a content-derived address from a shared neural map.Source: §1 · pp. 1–2
• The token address combines the unit address with the local RoPE rotation; causal order controls availability while attention scores select relevant content.Source: §1 · p. 2
• The paper proves adding units creates no new relative rotations and tests stable language-model behavior and retrieval across serialized facts.Source: §1 · p. 2
• Content-based addressing does not reduce attention cost or softmax competition, and its shared content map must generalize to new unit contents.Source: §1 · p. 2

• Each completed unit receives an orthogonal address from a shared neural map applied to its content, combined with the token's bounded local RoPE position.Source: §2 · p. 2
• The content descriptor is order-sensitive because embeddings are rotated by local position before pooling, so swapping differently embedded tokens can change the unit address.Source: §2 · p. 2
• Because the address and local-position rotations use the same fixed two-dimensional blocks, they commute and can be implemented by adding their angle vectors.Source: §2 · p. 2
• During causal generation, only completed units receive content addresses; the final incomplete unit uses the identity rotation because its descriptor is unavailable.Source: §2 · p. 2
• Within-unit relative operators remain ordinary RoPE, and inserting, deleting, or reordering other units leaves the relative operator between fixed tokens unchanged.Source: §2 · pp. 2–3
• For any number of units, relative operators stay within a fixed family that does not depend on unit count or serialized locations.By contrast, global RoPE's possible position differences grow with serialized context length.Source: §2 · p. 3
• The theorem does not guarantee generalization to arbitrary unseen content, although the shared map's parameter count does not grow with the number of units.Source: §2 · p. 3

Plain-text context growth

• The context-growth experiment used single-run, shallow models trained on 256-character chunks and evaluated at windows from 256 to 4096 characters.The four character-level Tiny Shakespeare models had four layers, four heads, and embedding dimension 128; the study was not a comparison among tuned long-context systems.Source: §3 · p. 3
• All-token perplexity changed from 4.81 to 4.86 for content addressing, versus 4.70 to 13.36 for continuous RoPE as context grew from 256 to 4096 characters.The content-addressed model’s context-growth curve remained approximately constant up to 16 times the training length.Source: §3 · p. 3

Retrieval from serialized facts

• Retrieval training used one fact per example, while evaluation used either the true fact alone or the true fact plus four distractors.The three-layer character model generated a sentence stating the target fact; chain2 and chain3 facts contained two and three entities, respectively, and hit@5 required every generated entity-name character to rank among the five highest-probability predictions.Source: §3 · pp. 3–4
• Content addressing had higher hit@5 in five of six test rows and tied or had lower PPL in all six; Table 2 reports test hit@5 and PPL on target entity-name characters.Continuous RoPE had higher hit@5 only in the single-fact chain2 row.Source: Table 2 · p. 4, §3 · p. 4

Related work

• Unlike CoPE and PaTH Attention, which accumulate changes through intervening tokens, direct content-based unit addresses leave comparisons between fixed tokens unchanged when unrelated units are added.CoPE and PaTH Attention are identified as retaining addresses obtained by accumulating changes through intervening tokens.Source: §4 · p. 4

Implementation

• Measured training overhead ranged from 10.3% at 0.8M parameters to 2.0% at 680M parameters.The pooling and projection add unit addresses to RoPE without changing attention heads or Transformer blocks.Source: §4 · p. 4

Discussion

• Content addresses do not encode ordinal distance, while attention remains finite and the shared map must generalize to new content; designers must choose unit boundaries.The experiments use lines and serialized facts, though units could also be sentences, paragraphs, chapters, documents, or books.Source: §4 · p. 4

5 Conclusion

• Content-based addressing preserves local RoPE and gives completed units content-based addresses, so adding units creates no new relative rotations.The construction was implemented in standard Transformer attention in the reported diagnostics.Source: §5 · p. 4

A Experimental Details

• Zero-initializing the final angle projection sets Re = I initially, preserving the original model before fine-tuning.Source: A Experimental Details · p. 5

A.1 Tiny Shakespeare

• Validation perplexity is measured over all tokens at context windows of 256, 512, 1024, 2048, and 4096 characters.The models are trained on one million characters using 256-character chunks and 5,000 iterations.Source: §A.1 · p. 5
• The four models share a 65-token vocabulary and four-layer, four-head architecture, with parameter counts ranging from approximately 810K to 819K.Training uses batch size 64, AdamW, and a learning rate of 3 × 10−4.Source: §A.1 · p. 5
• The content-addressed condition rotates embeddings within each newline-delimited line, mean-pools them, and maps the result to d/2 angles; the incomplete final line uses Re = I.The random-address condition instead samples an independent N(0, 1) angle vector for each completed line on every forward pass.Source: §A.1 · p. 5

A.2 Retrieval from Serialized Facts

• The dataset contains five domains, 134,935 facts, 25 chain2 relations, and 20 chain3 relations, yielding 6,116,520 sentences.Chain2 examples require generating one entity from another; chain3 examples require generating two entities from one.Source: §A.2 · p. 5
• Training loss covers every sequence token, but evaluation scores only target entity-name characters; hit@5 requires each scored character to rank among the top five predictions.Test rows use 100 names disjoint from training names.Source: §A.2 · p. 6
• Evaluation contrasts single-fact inputs with two multi-fact conditions: same entities under different relations, or the same relation with different entities.Training has only one fact per example, so it never requires choosing among multiple facts with the same relation.Source: §A.2 · p. 6
• The comparison includes continuous RoPE, content-based addressing, and two single-slope ALiBi models, using three-layer single-head causal Transformers.The content-addressed model resets local position at each fact and text boundary and assigns completed facts content-based addresses.Source: §A.2 · p. 6
• The content-addressed model leads on hit@5 in every multi-fact test row and five of six test rows overall.Table 3 reports complete retrieval results; higher hit@5 and lower perplexity are better.Source: Table 3 · p. 6, §A.2 · p. 6

B Timing Benchmark

• Each timing result averages ten timed training steps after one warmup step, with CUDA synchronization around the timed region.The benchmark uses an NVIDIA Tesla T4 and batch size 8 at 0.8M parameters; batch size is reduced for larger models to fit device memory.Source: B Timing Benchmark · p. 7
• Table 4 reports training-step timing at context length 1024, with overhead defined as the content-addressed model’s increase relative to baseline.Baseline time includes forward pass, backward pass, and optimizer update.Source: Table 4 · p. 7
• The added boundary detection, local rotation, pooling, and projection require O(Td) work for fixed hidden size, alongside Transformer work of order O(T 2d + Td2).The content-addressed model uses vectorized operations.Source: B Timing Benchmark · p. 7

C Continuity of Content-Derived Scores

• The proposition states that content-derived scores change continuously when the descriptor map φ is K-Lipschitz in operator norm and q and k have unit norm.The descriptors lie in a metric space and φ maps them to O(d).Source: C Continuity of Content-Derived Scores · p. 7
• The proof uses the triangle inequality, orthogonality of surrounding factors, and equality of the operator-norm differences for φ and its inverse.These ingredients are stated as the basis for the result.Source: C Continuity of Content-Derived Scores · p. 7
• The stability bound does not require or guarantee that distinct descriptors receive distinct addresses.Source: C Continuity of Content-Derived Scores · p. 7