Source-linked AI summary
Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory
Vakul Raghavan, Utkarsh Grover, Mingyang Mao, Chengwen Wang, Xiaomin Lin
TL;DR
Long-context models must retain and relate information in a compact memory, not merely process longer sequences. Double-Take addresses capacity, binding, and selective retention with multi-step updates, cross-position mixing, and bidirectional change-sensitive writes. Its memory remains informative beyond its 32 kb training context and transfers zero-shot from human to mouse.
Problem
As context grows, compact memory must retain associations, represent cross-position relationships, and preserve relevant structure; context length alone does not ensure usable long-range memory.
Source: §1 · p. 1Method
Results
DT remains informative more than 30× beyond its training context and transfers zero-shot from human to mouse.
Source: §1 · p. 2Takeaways & Limitations
The paper identifies capacity, binding, and selectivity as three requirements for usable long-range test-time memory.
Source: §5 · p. 10The study is limited to genomic sequence and lacks transcriptomic, epigenomic, or proteomic supervision; some recent TTT comparisons are constrained by unavailable reproducible implementations.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the $0.10$ chance floor; adding binding raises it to $0.92$. Trained only on $32\,\mathrm{kb}$ genomic contexts, DT reaches $0.927$ AUROC at $131\,\mathrm{kb}$ and $0.856$ near $1\,\mathrm{Mb}$, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.
1 INTRODUCTION
Long-context models must do more than process distant information: they must retain associations, connect positions, and preserve relevant structure. Double-Take (DT) addresses these needs with separate mechanisms for capacity, binding, and selective retention.
1 INTRODUCTION
- Long-context processing does not guarantee usable recall: access can depend on sequence position, and retrieval performance can degrade as context length and complexity increase.The introduction distinguishes processing distant information from remembering it.Source: §1 · p. 1
- Compressed memory raises three distinct requirements: retaining associations, binding events across positions, and keeping relevant relationships prominent as context grows.Genomics makes these requirements concrete because regulatory partners can be far apart while only a small part of the intervening sequence matters.Source: §1 · p. 1
- Large-chunk TTT writes can interfere with stored associations, self-associative writes lack cross-position relationships, and equal transport provides no mechanism for selective retention.The paper treats these as separate limitations: capacity, binding, and selectivity.Source: §1 · p. 2
- DT uses multi-step unpinned updates for capacity, local cross-position mixing for binding, and bidirectional change-sensitive writes for selective retention.The paper frames usable long-range memory as retaining enough information, relating positions, and preserving the relationships that matter.Source: §1 · p. 2
- DT remains informative more than 30× beyond its training context, recruits information outside the local receptive field through test-time memory, and transfers zero-shot from human to mouse.These are the introduction’s headline results.Source: §1 · p. 2
2 BACKGROUND
The background presents fast-weight memory and test-time training as ways to update a compact memory while processing a sequence, then motivates three questions about large-chunk writes. Genomic models make distant sequence increasingly accessible, leaving open whether memory preserves and connects the relevant information.
2.1 FAST-WEIGHT MEMORY AND TEST-TIME TRAINING
- Fast-weight memory writes temporary information into changing parameters; linear attention can be viewed as finite-state key–value storage, and TTT updates a small learnable model from incoming sequence.LaCT applies fast-weight updates to large token blocks, making each update responsible for many associations.Source: §2.1 · p. 2
- Large-chunk writes raise three questions: whether associations interfere, whether different positions can be connected, and which relationships should remain important.The paper names these requirements capacity, binding, and selectivity.Source: §2.1 · p. 2
2.2 LONG-CONTEXT GENOMIC MODELING
- Genomic models extend sequence access, but the background asks whether memory can preserve enough information, connect relevant positions, and retain important structure.The cited examples include DNA-based prediction of genome folding and regulatory prediction using long-range sequence.Source: §2.2 · p. 3
3 DOUBLE-TAKE
DT carries a persistent fast-weight state across chunks and changes large-chunk writes to address capacity, binding, and selectivity. Its mechanisms refine writes, mix nearby positions before writing, and use opposing context views to emphasize change.
3 DOUBLE-TAKE
- DT processes chunks with a persistent fast-weight memory, differing from LaCT through partial inner updates, causal cross-position mixing, and a bidirectional change-point write target.Each hidden representation supplies a query for reading, a key for addressing a write, and a value for the content written.Source: §3 · p. 3
3.1 PARTIAL UPDATES INCREASE CAPACITY
- A single large-chunk write superposes key–value associations, so overlapping keys retrieve cross-talk and capacity becomes limited as the number of pairs grows.DT uses multiple partial updates and carries the resulting state forward rather than replacing the information accumulated in earlier chunks.Source: §3.1 · pp. 3–4
3.2 LOCAL MIXING CREATES CROSS-POSITION BINDINGS
- Standard self-associative writes pair each key only with its same-position value, so additional updates cannot supply absent cross-position relationships.DT applies causal local mixing to query, key, and value streams before writing; local mixing creates bindings, while persistent memory transports them across chunks.Source: §3.2 · p. 4
3.3 CHANGE-POINT WRITES ADD SELECTIVITY
- DT writes the difference between left- and right-context summaries, so similar sides cancel while changes remain prominent.The forward and reversed views are processed together along the batch dimension, avoiding a second sequential traversal.Source: §3.3 · p. 5
4 EXPERIMENTS
The experiments test whether DT’s memory supports usable long-range information through controlled recall and genomic tasks. They evaluate context scaling, selective retention, and zero-shot transfer beyond human genomes.
4 EXPERIMENTS
- The evaluation combines controlled associative-recall tests with frozen-encoder genomic tasks and lightweight probes; the encoder is pretrained on 32 kb human genomic sequences.The model uses 87M parameters, 2048-token memory-update chunks, and a local attention receptive field of approximately ±1.5 kb.Source: §4 · p. 5
4.1 CAPACITY AND BINDING ENABLE LONG-RANGE RECALL
- Capacity plus binding raises cross-position recall from the 0.10 chance level to 0.92; at far separation, DT retains 0.827 recall while attention falls to 0.019.DT retains 0.93 recall at 32 associations, matching the memory-head dimension.Source: Figure 2 · p. 6, §4.1 · p. 6
4.2 DT USES CONTEXT BEYOND ITS TRAINING LENGTH
- DT + gain 4 outperforms the other approaches from 32 kb onward, peaks near 131 kb, and declines gradually toward 1 Mb.DT without gain also improves through 524 kb, beyond the 32 kb training length.Source: Figure 3 · p. 7, §4.2 · p. 7
4.3 DT RETRIEVES INFORMATION BEYOND THE LOCAL RECEPTIVE FIELD
- With adapting bidirectional memory, distal validation loss is 1.292, versus 15.30 with the same fast-weight state frozen; the bidirectional formulation beats the single-state control in all six paired evaluations.Targets were at least 3,072 bp from visible tokens, beyond the local path’s approximately ±1.5 kb receptive field; the frozen-state loss gap is treated as a mechanistic control.Source: §4.3 · p. 7
4.4 DT SELECTIVELY PRESERVES GENOMIC CHANGE
- The change-point write raises boundary-to-random ratios from 1.06 to 1.15 on controlled boundaries and 1.01 to 1.09 on gnomAD-SV; P(rare > common) rises from 0.42 to 0.67.The associative write remains close to no preference across distance.Source: Figure 4 · p. 8, §4.4 · pp. 8–9
- Frozen DT reaches mean AUROC 0.728 across nine GTEx tissues and 0.747 in matched whole blood versus 0.689 for frozen HyenaDNA, but enhancer–target assignment reverses the ordering, 0.628 versus 0.724.Published expert and CNN baselines use their own fine-tuning protocols, unlike the matched frozen-encoder comparison.Source: §4.4 · p. 8, Table 1 · p. 8
4.5 DT TRANSFERS BEYOND HUMAN GENOMES
- Under the matched logistic-probe protocol, mouse without adaptation, DT scores 0.786 AUROC at 32 kb and 0.785, 0.761, 0.783, and 0.796 from 131 kb to 1 Mb; Caduceus-PS reaches 0.773 at its 131 kb native limit.HyenaDNA-1m scores 0.733–0.710 from 131 kb to 1 Mb. Across mammals, centroid distance from human generally increases with evolutionary distance, while chicken departs from this trend.Source: Table 2 · p. 9, §4.5 · p. 9
5 CONCLUSION AND LIMITATIONS
Double-Take targets capacity, binding, and selectivity to make long-range test-time memory usable beyond its 32 kb training context. The study is limited to genomic sequence and lacks several forms of biological supervision; genomic results do not establish clinical or diagnostic validity.
5 CONCLUSION AND LIMITATIONS
- DT addresses capacity, binding, and selectivity through multi-step updates, cross-position mixing, and bidirectional change-point writes.Controlled experiments isolate the mechanisms, while genomic results show usefulness beyond the 32 kb training context and transfer across species at megabase scale.Source: §5 · p. 10
- The study is limited to genomic sequence and lacks transcriptomic, epigenomic, or proteomic supervision; some recent TTT comparisons are constrained by unavailable reproducible implementations.The authors identify extension across modalities, architectures, and domains as a next step.Source: §5 · p. 10
- Generative AI was used for language editing, code drafting and debugging, and LaTeX formatting, but not to generate measurements, determine results, or make scientific conclusions.Source: AI USE · p. 11
- The genomic experiments do not establish clinical or diagnostic validity, and applying DT elsewhere would require additional evaluation of data provenance, interpretation, and domain-specific risks.Source: ETHICS STATEMENT · p. 11
A DEFECTS IN STOCK TTT
Stock TTT has interference-limited capacity and self-associative writes that cannot express cross-position bindings. The appendix reports that multi-step unpinned updates improve recall, while local mixing makes cross-token binding expressible.
A DEFECTS IN STOCK TTT
- For orthonormal values and spherical keys, expected cross-talk energy relative to signal is (N −1)/d and grows linearly with load.Repeating plain steps without normalization only rescales the write, leaving the signal-to-cross-talk ratio unchanged.Source: A DEFECTS IN STOCK TTT · p. 13
- Stock renormalization progressively rotates the carried state toward each new write, whereas free retention dilutes rather than replaces the prior state.For nonlinear SwiGLU memory, the benefit of multiple steps is empirical.Source: A DEFECTS IN STOCK TTT · p. 13
- Self-associative writes cannot express cross-token retrieval: their cross-position coefficients are fixed by the key Gram matrix, regardless of inner rate or step count.A causal convolution of width c supplies local mixing for cross-token binding; at c = 1 it reduces to the self-associative case.Source: A DEFECTS IN STOCK TTT · pp. 13–14
- Recall reaches 0.99 at sixteen associations and 0.93 at thirty-two with five-step unpinned updates, versus stock recall of 0.60 at two associations.The memory saturates at approximately the head dimension of 32; the figure reports this result alongside the stock comparison in Table 8.Source: A DEFECTS IN STOCK TTT · p. 14
B DETAILED EXPERIMENTAL SETUP
- Downstream genomic evaluations generally freeze the pretrained encoder and train only a lightweight task-specific probe; memory modifications affect the forward-pass memory, not pretrained parameters.Source: B DETAILED EXPERIMENTAL SETUP · p. 14
B.1 MODEL ARCHITECTURE
The shared encoder combines local convolution, bidirectional sliding-window attention, fast-weight memory, and an MLP. DT modifies the same underlying memory to study capacity, binding, and selective retention.
B.1 MODEL ARCHITECTURE
- The 87M-parameter encoder has twelve prenormalized blocks, each containing depthwise convolution, bidirectional sliding-window attention, fast-weight memory, and a position-wise MLP.Its embedding width is 768, with 24 attention and memory heads of dimension 32.Source: §B.1 · p. 14
- Sliding-window attention communicates at most 128 positions per layer, for a maximum receptive field of ±1,536 nucleotides across twelve blocks.Information reaching a central readout from substantially beyond that range must use the fast-weight memory path.Source: §B.1 · p. 14
B.2 DT MEMORY CONFIGURATIONS
- The experiments modify the same fast-weight memory to separate capacity, cross-position binding, and selective retention without changing the surrounding encoder.Source: §B.2 · p. 15
- DT starts each chunk from the carried memory and performs five Newton–Schulz-normalized inner updates without projecting the resulting state back to a prescribed norm.The finite update budget limits each chunk’s movement of the persistent state while allowing current associations to be fitted more accurately; width-3 causal convolution exposes nearby relationships before writing.Source: §B.2 · p. 15
B.3 PRETRAINING DATA AND OBJECTIVE
The study pretrains its primary genomic model on 32,768-nucleotide human sequences with bidirectional masked-language modeling, then evaluates representations with lightweight readouts. Matched objective controls and external long-context models provide distinct comparison points.
B.3 PRETRAINING DATA AND OBJECTIVE
- Pretraining uses GRCh38 sequences of length 32,768 nucleotides, excluding chromosomes 8 and 9 from training and reserving them for downstream human evaluation.Source: §B.3 · p. 15
- The primary model uses bidirectional masked-language modeling, while span and distal masking serve as controls for whether explicit long-range supervision alone recovers DT-like behavior.Distal masking removes the local neighborhood around prediction targets, requiring information beyond the attention-only receptive field.Source: §B.3 · p. 15
B.4 COMPARISON MODELS
- Matched controls retain the 87M DT architecture, whereas public long-context models are external references with different architecture, scale, and pretraining.Because public models are substantially smaller, their comparison is not used to establish the causal effect of the memory modification.Source: §B.4 · p. 16
B.5 DOWNSTREAM READOUTS
- Genomic evaluations generally freeze pretrained encoder parameters and measure task performance with lightweight readouts trained on the representations.Source: §B.5 · p. 16
B.6 CONTROLLED MEMORY EXPERIMENTS
The controlled experiments isolate memory capacity, cross-position binding, and distance persistence; genomic context scaling then tests boundary discrimination beyond the pretraining context. The boundary task distinguishes annotated-boundary windows from distant controls, not calibrated genome-wide calls.
B.6 CONTROLLED MEMORY EXPERIMENTS
- Capacity tests vary key–value load up to 64 associations with a 32-dimensional memory head, comparing stock, five-step renormalized, and DT writes.Each method receives the same associations and queries; recall of the stored value is measured independently of cross-position binding.Source: §B.6 · p. 16
- Cross-position binding tests retrieval when a key and value occur at different positions, with ten candidate outputs and chance recall of 0.10.Width-3 mixing tests whether the missing relationship must be represented before entering memory.Source: §B.6 · pp. 16–17
- Distance tests increase the separation of associated positions while holding local mixing fixed, comparing DT with sliding-window attention to assess fast-weight transport.The local convolution cannot itself span the far separation.Source: §B.6 · p. 17
B.7 TAD-BOUNDARY CONTEXT SCALING
- The context sweep evaluates frozen DT from 2 kb to approximately 1 Mb, so contexts beyond 32 kb test inference-time extrapolation.Human evaluation uses held-out chromosomes 8 and 9 and balanced boundary-centered versus distant-control windows.Source: §B.7 · p. 17
- The boundary task measures discrimination between boundary-centered windows and windows far from boundaries, not calibrated genome-wide boundary calling.Intermediate-distance positions are absent, limiting the absolute interpretation of AUROC but not comparisons on identical examples.Source: §B.7 · p. 17
B.8 DNALONGBENCH VARIANT-EFFECT EVALUATION
The evaluation pairs variant-effect prediction with enhancer–target-gene assignment to test frozen representations on regulatory change and distant-element identity. Published task-specific model scores provide scale references, while frozen-DT versus frozen-HyenaDNA in whole blood is the protocol-matched comparison.
DNALONGBENCH VARIANT-EFFECT EVALUATION
- eQTL evaluation asks whether an allele replacement changes regulatory state, using frozen DT with a lightweight logistic adapter across nine GTEx tissues.Whole blood also provides a protocol-matched comparison against frozen HyenaDNA.Source: §B.8 · p. 17
- Published expert, CNN, and fine-tuned model results establish benchmark scale, not protocol-matched or causal comparisons with DT.The frozen-DT versus frozen-HyenaDNA whole-blood result is the protocol-matched encoder comparison.Source: §B.8 · pp. 17–18
- Enhancer–target-gene assignment asks which distant enhancer is associated with a gene, testing preservation of long-range element identity.It complements eQTL effect prediction by contrasting change-sensitive representation with precise distant-element identity.Source: §B.8 · p. 18
B.9 STRUCTURAL PERTURBATION EXPERIMENTS
The experiments perturb boundary sequence and compare distant effects of boundary-centered versus matched random perturbations. They test both whether the readout responds to boundary-scale sequence and whether transport selectively favors boundary information.
B.9 STRUCTURAL PERTURBATION EXPERIMENTS
- 0.277, 0.224, and 0.210 are the boundary-probability drops after centered 5, 20, and 50 kb deletions, respectively.The 100 annotated boundaries are evaluated at 262 kb context against intact windows.Source: §B.9 · p. 18
B.10 SELECTIVE LONG-RANGE TRANSPORT
- 0.99, 1.07, and 1.07 are the boundary-to-random effect ratios for associative writes at 16, 32, and 65 kb, respectively, showing little preference.The distal readout is fixed and perturbations occur outside its local attention receptive field; Ssel > 1 denotes preferential boundary transport.Source: §B.10 · p. 18
- 0.67 versus 0.42 is the probability that a rare constrained deletion produces the larger propagated effect under change-point versus associative writes.This compares constrained rare deletions with common tolerated variants.Source: §B.10 · p. 18
C EXTENDED EXPERIMENTAL DETAILS
- The extended protocol section situates these tests within a progression from controlled memory experiments through genomic extrapolation and selective transport.It describes experiment construction, fixed conditions, and the comparison each test measures.Source: C EXTENDED EXPERIMENTAL DETAILS · p. 18
C.1 CAPACITY AND BINDING EXPERIMENTS
Controlled recall tests distinguish capacity, binding, and distance persistence, followed by genomic evaluations of extrapolation, selective transport, and cross-species transfer. The results show that these memory properties are tested separately rather than inferred from context length alone.
C.1 CAPACITY AND BINDING EXPERIMENTS
- 0.93 recall at 32 associations is retained by DT, which remains near perfect through 16, while repeated renormalized updates degrade near the head dimension.The memory-head dimension is d = 32.Source: Table 8 · p. 19, §C.1 · p. 19
- 0.92 recall follows cross-position mixing, whereas capacity repair alone leaves the ten-choice task at chance, 0.10.The added width-3 mixing makes the relationship representable before the write.Source: Table 9 · p. 19, §C.1 · p. 19
- DT changes from 0.859 to 0.827 across near and far separations, while local attention falls from 0.142 to 0.019.The comparison tests whether locally created associations persist through the fast-weight memory beyond the local operator's reach.Source: §C.1 · p. 19, Table 10 · p. 20
C.2 LONG-CONTEXT GENOMIC EXTRAPOLATION
- The boundary context sweep extends from 2 kb to approximately 1 Mb with fixed pretrained parameters, making evaluations beyond 32 kb inference-time extrapolation.It discriminates boundary-centered windows from distant controls, not calibrated genome-wide boundary calls because intermediate-distance positions are omitted.Source: §C.2 · p. 20
C.3.1 VARIANT-EFFECT AND LONG-RANGE IDENTITY TASKS
- The frozen-encoder tasks pair eQTL effect prediction with enhancer–target-gene identity; published fine-tuned model scores are external references, not causal comparisons.The whole-blood frozen-DT versus frozen-HyenaDNA comparison is protocol matched.Source: §C.3.1 · p. 21
C.3.2 BOUNDARY DELETION
- The centered 5 kb boundary deletion produces the largest boundary-readout change; larger 20 and 50 kb deletions have similar effects.This is a perturbational test of boundary-scale sequence at 262 kb context.Source: §C.3.2 · p. 21
C.3.3 SELECTIVE LONG-RANGE TRANSPORT
- The change-point write increases selectivity in controlled and real structural-variant settings; rare constrained deletions more often yield the larger propagated effect than common variants.The reported probability rises from 0.42 under associative writes to 0.67 under change-point writes.Source: §C.3.3 · p. 22, §C.3.4 · p. 22
C.4 CROSS-SPECIES TRANSFER
- Human TAD-boundary evaluation uses held-out chromosomes, while mouse uses a human-pretrained encoder without adaptation; frozen and adapted values align across mammals but chicken departs from that pattern.The broader sweep compares chimpanzee, macaque, mouse, and chicken in increasing evolutionary distance from human.Source: §C.4.1 · p. 22, §C.4.2 · pp. 22–23