Source-linked AI summary
What Attention Recalls and Recurrence Controls in Hybrid Language Models
Kirill Afendulev, Alexey Dontsov, Elena Tutubalina, Anton Korznikov
TL;DR
Hybrid language models combine attention with fixed-size recurrent state, but what each channel contributes at inference is unclear. Cache interventions show that exact retrieval follows attention while output language and persona follow recurrence; state-swap assigns retrieved values and language to their respective cache sources.
Problem
In hybrid language models, what does each component carry at inference?
Source: §1 · p. 1Method
The paper uses split-prefill to retain one context cache channel at a time and state-swap to pair caches from different contexts.
Source: §1 · p. 1Results
64–98% of baseline retrieval accuracy survives in KV-ONLY while REC-ONLY collapses to zero; language and persona instead favor recurrence over KV-ONLY.
Source: §4 · pp. 2–3Takeaways & Limitations
The KV cache provides addressable retrieval, while the recurrent state shapes language, persona, and semantic field of generation.
Source: §6 · p. 4Split-prefill creates a partly out-of-distribution cache combination, so some channel-only collapse could reflect broken coherence; state-swap tests a closer-to-natural setting.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.
1 Introduction
Hybrid language models pair softmax attention with fixed-size recurrent blocks, but what each channel carries during inference is unclear. Cache-level interventions reveal a functional split: attention supports exact retrieval, while recurrence preserves response policy.
1 Introduction
- Split-prefill keeps one context channel and drops the other; state-swap crosses the two caches from different contexts.The interventions are instantiated on Qwen3.5-4B and Falcon-H1-3B-Instruct.Source: §1 · p. 1
- The recurrent state collapses to zero on exact retrieval, while the two components separate behaviorally under the interventions.The authors distinguish this functional dissociation from the partial recall reported for stand-alone state-space and linear-attention models (Jelassi et al., 2024; Park et al., 2024).Source: §1 · p. 1
2 Related Work
Hybrid models combine attention with fixed-state blocks, while earlier work links compressed-state models to recall difficulty. This paper situates its approach among hybrid architectures and causal component analysis.
2 Related Work
- Recent open-weight hybrids combine standard attention with compressed-state blocks, using interleaved, 3:1, or parallel-hybrid designs.The cited examples are Jamba, Qwen3.5, and Falcon-H1, respectively.Source: §2 · p. 1
- Prior work reports exact-copying and retrieval weaknesses in fixed-state models; this paper claims the recurrent state is not addressable memory for retrieval at inference.The passage situates the claim alongside prior studies (Jelassi et al., 2024; Park et al., 2024; Hsieh et al., 2024).Source: §2 · p. 2
- Mechanistic analyses have mapped transformer circuits and activation patching, with Sharma et al. (2024) extending this line to Mamba.The paper complements that work by causally swapping two components of a hybrid on the same prompt.Source: §2 · p. 2
3 Method
The method separates context carry-over into attention and recurrent channels, then tests each channel alone or pairs channels from different contexts.
3 Method
- Split-prefill populates the context’s attention KV cache and recurrent state, then processes the query with both channels kept, only KV kept, or only recurrence kept.Because the query is processed normally, the intervention isolates carry-over from the context in a specific channel.Source: §3 · p. 2
- State-swap prefills two same-format, different-content contexts, then runs a shared query using one context’s KV cache and the other’s recurrent state.The mirror pairing tests which context each answer property follows.Source: §3 · p. 2
- Implementation details are provided in Appendix A.Source: §3 · p. 2
4 Quantitative Dissociation
Four isolated diagnostic tasks reveal opposing channel patterns: attention preserves exact retrieval, while recurrence preserves response mode. State-swap results further attribute retrieved values and output language to different cache sources.
4 Quantitative Dissociation
- Small synthetic tasks isolate one capability each, addressing the attribution problem of long-context benchmarks that demand retrieval and response-mode maintenance together.The authors state that the large reported effects offset the trade-off of using isolated tasks.Source: §4 · p. 2
- 64–98% of baseline retrieval accuracy survives in KV-ONLY, while REC-ONLY retrieval collapses to zero across both architectures.The main-result figure presents the four diagnostic tasks relative to the full-model baseline; the table caption identifies retrieval as surviving only through KV.Source: §4 · p. 2, Figure 2 · p. 3, Table 1 · p. 3
- 70–80% of full-model language-following accuracy survives in REC-ONLY, while KV-ONLY drops to ∼1%, a 60–80× gap.Persona follows the same direction, with REC-ONLY 3–5× higher than KV-ONLY.Source: §4 · p. 3, Figure 2 · p. 3
- The KV cache contains the language instruction, but the model does not act on it when recurrence is dropped.The passage attributes control of the target continuation mode to the recurrent state.Source: §4 · p. 3
5 Four Causal Case Studies
Four causal case studies use mixed caches and false-recall probes to distinguish what attention retrieves from what the recurrent state contributes.
5.1 Case Study #1: Mixed Memories
- ≥99% of mixed-cache items are jointly correct in both models: KV supplies cA’s value and recurrence cB’s language; same-source controls match FULL.The mixed cache pairs KV from cA with recurrent state from cB in one forward pass.Source: §5.1 · p. 3, Figure 3 · p. 3
5.2 Case Study #2: Recall, but Not Control
- KV-ONLY names the requested language but answers in English, while REC-ONLY reverses the pattern on behavioral queries.The case study pairs a language-label query with a behavioral query.Source: §5.2 · p. 4, Figure 4 · p. 4
5.3 Case Study #3: Associative False Recall
- REC-ONLY sharply raises false-yes responses to associated absent targets while neutral-absent controls stay near floor; KV-ONLY rejects the trap.The recurrent state preserves the semantic field of seen words, making an untokenized target seem familiar.Source: §5.3 · p. 4, Figure 5 · p. 4
5.4 Case Study #4: Conjunction Trap
- REC-ONLY sharply raises false-yes rates for unseen conjunctions but stays near floor when only one morpheme is present, indicating lost item binding.Component features survive in recurrence, but the binding that ties them to a specific item is lost.Source: §5.4 · p. 4, Figure 6 · p. 4
6 Conclusion
The conclusion distinguishes the channels by how the model accesses their contents, not by how long they retain context. It frames the recurrent state as a different kind of memory and cautions that combined lookup and behavioral-conditioning benchmarks cannot reveal this distinction.
6 Conclusion
- The two components do not differ in how long they retain past context; they differ in how the model accesses what they hold.The conclusion describes this as a functional distinction.Source: §6 · p. 4
- KV is an addressable store for specific items, while the recurrent state is a compressed prior shaping language, persona, and semantic field.The claim concerns each channel’s causal contribution to generation, not what it statically encodes.Source: §6 · p. 4
- State-swap attributes the two channels’ contributions causally at the level of a single answer.The conclusion argues that benchmarks scoring addressable lookup and behavioral conditioning together cannot show this distinction.Source: §6 · p. 4
Limitations
The authors limit their claim to causal contributions at inference, note that split-prefill may create incoherent cache pairs, and leave static-content probing for future work.
Limitations
- The claim concerns what each channel causally contributes at inference, not its static contents, and neither channel is claimed to lack the other type of content.The KV cache can name the requested language on demand even when KV-only cannot make the model answer in that language.Source: Limitations · p. 5
- Mapping static information in the two channels remains future work and would require trained probing classifiers on activations from each channel.Source: Limitations · p. 5
A Implementation Details
- Caches are extracted after context prefill and before any query token; Qwen3.5-4B uses the HuggingFace Gated DeltaNet integration, while Falcon-H1 uses its official HuggingFace release.Source: A Implementation Details · p. 6
B Split-prefill: Detailed Procedure
- The split-prefill setup reuses caches from one context prefill, keeps the query byte-identical, and sets its position from the retained KV cache.REC-ONLY starts the query at position 0; padding absent KV with zero keys would introduce another attended condition.Source: B Split-prefill: Detailed Procedure · p. 7
- State-swap pairs KV and recurrent caches from contexts differing in retrieved content and response mode, then runs a shared query; same-source assemblies provide controls.The procedure asserts identical query rendering and aligns positions to the KV-source context length when required.Source: B Split-prefill: Detailed Procedure · p. 7
C Datasets
- Dataset tasks include dictionary retrieval, list indexing, language following, and persona matching, each with task-specific queries or scoring.Retrieval uses short ASCII keys and values; language following uses an answer-only-in-L instruction with a meaning-preserving English question, and persona match is judged by an independent model.Source: C Datasets · p. 8
- The conjunction trap tests unseen combinations of list items, while association lists are paired with neutral and positive-present controls.Conjunction controls include targets unrelated to list items, targets sharing one component morpheme, and the listed conjunction itself.Source: C Datasets · p. 8
D Quantitative Dissociation: Full Results
Across four models, recurrent-only retrieval collapses while recurrent-only behavioral performance generally exceeds KV-only; a Qwen-specific layer sweep finds retrieval needs full KV and behavioral transfer needs full recurrence.
D Quantitative Dissociation: Full Results
- 0/600 recurrent-only retrieval on every seed for every model; recurrent-only behavioral cells exceeded KV-only except Falcon-H1-7B persona (KV-only 0.29, REC-ONLY 0.25; −3.8 p.p. [−6.9, −0.6]).Source: D Quantitative Dissociation: Full… · p. 8
- KV-only retrieval accuracy improves monotonically with model size across both families, while recurrent-only retrieval collapse persists.Source: D Quantitative Dissociation: Full… · p. 8
- Retrieval is preserved only when KV is full, while behavioral transfer is preserved whenever recurrent state is full; these layer-wise sweep modes were tested only on Qwen.Source: D Quantitative Dissociation: Full… · p. 8
E Demonstration 1 — State-swap
State-swap tests whether answer properties follow the KV cache or recurrent state when the two come from different contexts. A language-label probe separately contrasts naming a language with behaving in it.
E Demonstration 1 — State-swap
- On Qwen-4B and Qwen-9B, every crossed example is jointly correct; Falcon-H1-3B is at least 0.99 both-correct, while Falcon-H1-7B retains perfect value retrieval but reaches 0.87–0.89 both-correct.Same-source controls are 1.00 both-correct for Qwen-4B, Qwen-9B, and Falcon-H1-7B, and approximately 0.99 for Falcon-H1-3B.Source: E Demonstration 1 — State-swap · pp. 9–10
- Each state-swap crossing combines the KV cache from one context with the recurrent state from another, then uses a shared query to assess which source each answer property follows.The two crossing directions are KA + RB and KB + RA, with matched same-source controls.Source: E Demonstration 1 — State-swap · p. 8, Figure 7 · p. 10
F Demonstration 2 — Label vs. behavioural language
- The self-report probe distinguishes an English language name from the name written in the target language, the latter requiring the model to enact the language behavior.It uses 20 languages, with 20 examples per seed across three seeds.Source: F Demonstration 2 — Label vs.… · p. 10
- For Qwen3.5-9B, KV-ONLY gives the English language name 90% of the time but the target-language label 0.05; REC-ONLY reverses this at 0.10 English and 0.95 target-language.Qwen3.5-4B and Falcon-H1-7B show the same direction.Source: F Demonstration 2 — Label vs.… · p. 10
G Demonstration 3 — Associative (DRM) trap
The associative trap tests whether recurrent-only generation treats a contextually associated but absent word as present. Neutral-absent and present-word controls help distinguish associative false alarms from yes-bias and confirm true membership responses.
G Demonstration 3 — Associative (DRM) trap
- The task asks whether an exact target word appeared in a list, comparing associated-absent and neutral-absent lists with a positive-present condition.The false-positive rate is the fraction of “yes” responses on absent-target conditions; the true-positive rate is the fraction on the present condition.Source: G Demonstration 3 — Associative (DRM)… · p. 10
- REC-ONLY associated-absent false-positive rates are 0.87 for Qwen-4B, 0.89 for Qwen-9B, 0.45 for Falcon-H1-3B, and 0.99 for Falcon-H1-7B.For three of four models, the neutral-control false-positive rate stays at or below 0.10; Falcon-H1-7B is the exception, with a neutral rate of 0.18.Source: G Demonstration 3 — Associative (DRM)… · pp. 10–11
- Falcon-H1-7B’s associated response is less cleanly attributable to association because its KV-ONLY associated and neutral false-positive rates are both above 0.55.The recurrent-only associated rate exceeds its neutral rate by 0.81 p.p., but the authors advise a more guarded interpretation for this model.Source: G Demonstration 3 — Associative (DRM)… · pp. 10–11
H Demonstration 4 — Memory-conjunction trap
The memory-conjunction trap tests whether recurrent-only generation treats an unseen compound as present when both of its component morphemes appeared in the list. Matched single-part and neutral controls help separate this signal from general false-positive susceptibility.
H Demonstration 4 — Memory-conjunction trap
- Each probe is an unseen compound whose two morphemes both occur in a list; controls include neutral-absent, left-only, right-only, and positive-present items.For example, the probe “sunlight” can be paired with “sunflower” and “moonlight.”Source: H Demonstration 4 — Memory-conjunction… · pp. 11–12
- REC-ONLY conjunction-absent false-positive rates exceed both single-part-absent rates on all four models: Qwen-4B 0.32 vs. 0.18/0.05; Qwen-9B 0.17 vs. 0.08/0.03; Falcon-H1-3B 0.15 vs. 0.06/0.04; Falcon-H1-7B 0.75 vs. 0.61/0.48.Falcon-H1-7B also has a neutral false-positive rate of 0.68, so its absolute rates are inflated even though the conjunction effect persists relatively.Source: H Demonstration 4 — Memory-conjunction… · p. 12
- The comparison tests whether the conjunction-absent response exceeds each single-part-absent response, while neutral-absent and positive-present rates provide additional controls.Table 7 reports five rates per model and mode.Source: H Demonstration 4 — Memory-conjunction… · p. 12, Table 7 · p. 13