PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size

MyungHoon Ryu, XinYu Piao, Jong-Kook Kim

arXiv:2609.12686v1cs.AIcs.CL

TL;DR

Long-context QA requires recall beyond pretrained context windows, while token-level memory grows with input length and existing training-free approaches can lose needed details or retain source-sized archives. The method encodes facts from bounded document windows as activation-indexed residual vectors outside the frozen model and uses query-conditioned feed-forward gating to reconstruct selected facts after source removal. At two million tokens, it achieves 50–80% single-fact accuracy and exceeds every evaluated training-free method on all three models, with stored fact memory under one gigabyte in every measured condition.

  • Problem

    Long-context QA requires recall beyond pretrained context windows, while token-level memory grows with input length and existing training-free approaches can lose needed details or retain source-sized archives.

    Source: §1 · p. 1

  • Method

    The method encodes facts from bounded document windows as activation-indexed residual vectors outside the frozen model and uses query-conditioned feed-forward gating to reconstruct selected facts after source removal.

    Source: §1 · p. 1

  • Results

    At two million tokens, it achieves 50–80% single-fact accuracy and exceeds every evaluated training-free method on all three models, with stored fact memory under one gigabyte in every measured condition.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Selective factual recall is the main benefit of bounded activation memory, while broad synthesis, implicit relations, and unstored context remain better suited to raw-text retrieval or full-context processing when feasible.

    Source: §6 · p. 10

    Multi-fact composition, encyclopedic recall, and questions with limited correspondence to stored anchors remain constraints on selective recall.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) process long contexts, including long documents and lengthy conversations, but face token-level memory usage that increases proportionally to input length. Although model optimization and lossy prompt compression are widely used, these methods still fail to solve the long-context recall problem beyond pretrained and size-constrained context windows. This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases, without additional training. The main idea is to reconstruct facts using parameter activations in the LLM's feed-forward layers, which store residual vectors representing facts from the source document. Utilizing residual vectors allows the LLM to deterministically reconstruct query relevant facts without referencing the original document, preserving high fidelity and reducing memory usage without fine-tuning weights. Experimental results show that the proposed method enables answering single-fact questions in two-million-token story contexts where previous methods fail.

1 Introduction

Long-context recall remains constrained by context-window limits, memory growth, and losses from methods that discard or selectively retrieve earlier context. The paper proposes selective factual recall through external residual vectors and reports strong single-fact accuracy at two million tokens.

1 Introduction

  • Long-context QA requires recalling evidence beyond pretrained context windows, while token-level memory grows with input length and models miss or fail to connect distant evidence.Long interactions may span turns, sessions, and source files, and users expect continuity across them.Source: §1 · p. 1
  • Context extension requires additional optimization, while training-free compression and retrieval can discard needed details or incur source-length-dependent archives.The paper identifies these limitations as degrading long-context QA beyond pretrained context windows.Source: §1 · p. 1
  • The method converts bounded document windows into query-agnostic fact records, then reconstructs selected records after source removal using activation-indexed residual vectors without editing model weights.The overview describes gated injection at assigned feed-forward layers during response generation.Source: §1 · p. 1, Figure 1 · p. 2
  • At two million tokens, the method reaches 50–80% single-fact accuracy on all three models, exceeds every evaluated training-free method, and stores facts in less than one gigabyte in every measured condition.The compared training-free methods fall to zero or near-zero accuracy at this length.Source: §1 · p. 2

2 Related Work

Prior work extends context length, edits model knowledge, reduces retained context, or retrieves source segments, each with distinct trade-offs. This method instead stores selected facts in external residual vectors while keeping model weights fixed.

2 Related Work

  • Position scaling and long-context fine-tuning extend supported input length through additional optimization, while document-specific adaptation internalizes a target document in model parameters.Longer supported inputs still produce token-level inference state that grows with input length.Source: §2 · p. 2
  • Truncation, streaming, cache methods, and prompt compression restrict or shorten retained context, and query-agnostic reductions can remove evidence needed by later questions.Retrieval preserves source text but answer quality depends on retrieving every required segment, while its archive and index grow with source size.Source: §2 · p. 2
  • The proposed method optimizes external residual vectors for selected facts while keeping model weights fixed, and removing the external store restores the baseline model.It retains sentence-level fact records selected before future queries arrive, without the original document.Source: §2 · p. 2

3 Proposed Method

The method stores bounded, query-agnostic fact records as activation-keyed residual vectors while keeping the language model frozen. After source removal, query routing and gated recall reconstruct selected facts for answer generation.

3.1 Overview

  • The frozen model encodes each retained fact as an external residual vector keyed by a feed-forward activation, then reconstructs selected facts after source tokens and their KV cache are removed.Retained state follows the fact budget rather than source length.Source: §3.1 · pp. 2–3

3.2 Memorization Phase

  • Memorization extracts query-agnostic anchor–value records, retains them under a fixed budget, assigns them round-robin to selected layers, and optimizes residuals to regenerate their stored sentences.Recall prompts bind each fact key to an activation at its assigned layer; residual vectors are optimized while model parameters remain fixed.Source: §3.2 · p. 3

3.3 Response Phase

  • At response time, anchor rankings route questions to indexed facts; a similarity gate injects matching residuals to reconstruct stored sentences, which are ordered by source offset for answer generation.Routing combines dense semantic and lexical rankings with reciprocal-rank fusion; the gate opens when the strongest similarity exceeds its threshold.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4 Evaluation

The evaluation compares long-context recall and composition, encyclopedic-prose recall, and memory use. Results show strong single-fact recall at long inputs, alongside model-dependent composition and prose performance.

4.1 Experimental Setup

  • Evaluation uses three frozen instruction-tuned models with distinct context windows and activation spaces, testing BABILong factual recall and composition plus RULER QA in encyclopedic prose.BABILong uses QA1 and QA2 length sweeps and a QA1-to-QA5 task grid; RULER QA places SQuAD evidence in distractor passages.Source: §4.1 · pp. 5–6

4.2 Long-Context Recall and Composition

  • At 2M, the proposed method scores 50% on Phi-3.5-mini, 67% on Qwen3-4B, and 80% on Llama-3.1-8B, exceeding every plotted method.The advantage emerges at long inputs; at 128K and longer, the method remains nonzero in every measured cell.Source: Figure 4 · p. 6, §4.2 · p. 7
  • At 1M, only Phi-3.5-mini answers a two-fact question, scoring 50% on two questions; Llama-3.1-8B falls to zero after 128K.Single-fact recall therefore does not imply successful composition.Source: §4.2 · p. 7

4.3 Recall in Encyclopedic Prose

  • In encyclopedic prose, the method leads comparisons for Phi-3.5-mini through 2M and Llama-3.1-8B through 1M, but Qwen3-4B ties at 2M.Phi-3.5-mini's 100% scores at 1M and 2M each represent one question.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Memory Scaling

  • From 4K to 2M, measured peak memory stays at 9.3–10.3 GB on smaller models and 17.1–17.9 GB on Llama-3.1-8B, unlike growing full attention.The activation store grows by less than 0.8 GB; Chunk RAG uses 7.4–8.9 GB more than truncation at 2M.Source: §4.4 · pp. 7–8, Figure 5 · p. 8
  • Table 3 classifies recorded failures as extraction, routing, reconstruction, or answer-model failures, while retaining unattributed cases in n and accuracy.Its BABILong rows cover 128K to 2M, and RULER 16K is separated from longer inputs.Source: Table 3 · p. 8

5 Ablation Studies

The ablations show that recall depends on task demands, fact representation, layer allocation, evidence position, and routing. Direct factual recall can remain strong across long inputs, while composition and question-to-anchor matching expose limits.

5.1 Performance by Reasoning Requirement

  • QA1 reaches 100% at 128K and 256K, but QA3 and QA4 remain near zero from 16K; QA4 loses evidence in extraction, while QA3 fails despite full coverage.QA3 reconstruction fidelity is at least 0.77, and QA5 remains model dependent.Source: §5.1 · p. 8

5.2 Reconstruction Capacity and Layer Allocation

  • Template recall stays between 93.6 and 98.5% from 128 to 4,096 facts, whereas shared narrative recall and fidelity decline overall with store size.At 4,096 facts, Llama-3.1-8B has the highest fidelity and Qwen3-4B the highest recall EM.Source: §5.2 · p. 8, Figure 7 · p. 10
  • The compound anchor at 2M recovers the relevant sentence, while the generic anchor at 1M repeats despite twice the per-layer load; the conditions also differ in storage load.The comparison does not isolate anchor specificity from input length and storage load.Source: §5.2 · p. 8, Table 4 · p. 9
  • Increasing K from 1 to 6 raises accuracy from 62.5 to 87.5% while routing recall stays at 100%; the K=12 result uses a different split.The paired development comparison locates the observed improvement after routing.Source: §5.2 · p. 9

5.4 Encoding and Deletion

  • On five Qwen3-4B questions, reducing the optimization schedule lowers fidelity from 0.99 to 0.87 but leaves EM at 60%; deleting eight of 64 entries leaves all 56 retained outputs unchanged.Deleted-fact recall falls to zero on each tested model.Source: §5.4 · p. 9

5.5 Routing and Gating

  • The NoLiMa-derived probe answers one of eight questions at both retention budgets, versus seven of eight for Chunk RAG, exposing difficult question-to-anchor routing.The proposed method also fails on all four one-hop questions; a larger retention budget does not resolve the observed access limitation.Source: §5.5 · p. 9

5.3 Sensitivity to Evidence Position

  • 100% accuracy holds at all five tested source positions and three lengths, including 2M, while competing methods show position-sensitive or lower long-input recall.At 2M, Chunk RAG varies little by position but reaches only 72–73%.Source: §5.3 · p. 9, Figure 8 · p. 10

5.5 Routing and Gating

  • Lowering θ from 0.75 to 0.45 cuts Llama-3.1-8B accuracy from 80 to 40% and fidelity from 1.00 to 0.62, while Qwen3-4B accuracy is unchanged; dense-only rows lack a matched hybrid reference.The dense-only rows therefore do not establish a benefit from reciprocal-rank fusion.Source: §5.5 · pp. 9–10

6 Discussion and Limitations

The method’s main benefit is selective factual recall, but composition, encyclopedic recall, and weak question-to-anchor correspondence constrain its scope. The evaluation also leaves runtime costs and several generalizability and user-trust questions unresolved.

6 Discussion and Limitations

  • Selective factual recall is the main benefit, while multi-fact composition and encyclopedic recall vary by model and question; broad synthesis and unstored context remain better suited to raw-text retrieval or full-context processing when feasible.The NoLiMa probe identifies anchor routing as a limitation when questions have limited lexical or semantic correspondence to retained anchors.Source: §6 · p. 10
  • Encoding requires O(N) fixed-step optimization loops, and each response routes across N entries and decodes routed facts separately before the answer pass.Encoding cost amortizes only when later questions reuse retained entries.Source: §6 · p. 10
  • Evaluation covers three model families up to 8B parameters, one GPU class, and English automated benchmarks; million-token scores use one to five samples and indicate feasibility, not precise accuracy.Human feedback, trust, recovery after recall failure, and longitudinal use remain unmeasured; bitwise reproducibility across hardware or numerical configurations is not established.Source: §6 · pp. 10–11

7 Conclusion

The method recalls facts from long contexts without additional training or model-weight editing. At two million tokens, it answers at least half of single-fact questions for all three models, while compared training-free methods fall to zero or near zero.

7 Conclusion

  • At two million tokens, the method answers at least half of single-fact questions for all three models, while every compared training-free method falls to zero or near zero.The method’s recall remains unchanged at every tested source location, and response-time GPU memory stays close to the frozen model alone.Source: §7 · p. 11
  • Repeated access to a stored entry returns the same fact, and deleting an entry leaves answers from remaining facts unchanged in the deletion test.The paper frames this as reproducible recall and selective fact removal for long-running user and agent interactions.Source: §7 · p. 11
  • The method recalls past facts by reconstructing residual vectors assigned to feed-forward layers, without additional training or model-weight editing.It stores facts from long contexts and answers without the original document.Source: §7 · p. 11
  • Stored entries occupy less space than the KV cache they replace, while response-time GPU memory remains close to that of the frozen model alone.The paper identifies broader relational coverage, stronger semantic routing, and evaluation of privacy and interactive use as future work.Source: §7 · p. 11

Generative AI Disclosure

Anthropic Claude assisted with code implementation, while Claude and OpenAI Codex assisted with manuscript revision; the authors reviewed all AI-assisted content and remain responsible for the work.

Generative AI Disclosure

  • Anthropic Claude assisted in implementing the research ideas in code.The disclosure separately names Claude and OpenAI Codex as manuscript-revision assistants.Source: Generative AI Disclosure · p. 11
  • Anthropic Claude and OpenAI Codex assisted in revising the author-written manuscript.The disclosure says the authors reviewed all AI-assisted content.Source: Generative AI Disclosure · p. 11
  • The authors reviewed all AI-assisted content and remain responsible for the work.Source: Generative AI Disclosure · p. 11

Ethics and Privacy Statement

The stored memory can retain sensitive statements after source disposal. The evaluation uses public benchmark corpora and no human subjects, while deployment risks and safeguards remain relevant.

Ethics and Privacy Statement

Loading 2609.12686v1…