PaperlayerView exact revision on arXiv

Source-linked AI summary

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Chaogui Gou

arXiv:2610.21165v1cs.CL

TL;DR

Long-context inference faces computational and memory costs, while long inputs can mix useful evidence with repetition, outdated information, and noise. This survey organizes compression as lifecycle management across memory states and state transitions, with a shared budgeted cost model. It concludes that the central issue is deciding which information participates in inference, when, and in what form.

  • Problem

    Long inputs increase inference and KV-cache costs while mixing key evidence with repetition, outdated states, and noise, so systems must manage information under finite budgets.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The survey distinguishes compute-active, compute-inactive, and intermediate memory states, then models memory operations as lifecycle transitions under a budgeted cost model.

    Source: §1 · p. 2

  • Results

    The survey concludes that long-context compression is a problem of managing which information participates in inference, when, and in what form, rather than simply reducing length.

    Source: §9 · p. 20

  • Takeaways & Limitations

    Future systems should move toward dynamic, joint, and governable memory management with tiered allocation, verifiable forgetting, and full-lifecycle cost accounting.

    Source: §9 · p. 20

    The lifecycle feedback loop can preserve errors: write-back may store erroneous outputs or low-quality reflections that are retrieved in later tasks.

    Source: §6.5 · p. 16

Abstract

from arXiv · show

Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model that places these operations in a single cost account. The resulting framework unifies short-term context compression, long-term memory management, and the transitions between them, and shows that the central challenge is not merely reducing information volume but deciding which information should participate in costly inference, when, and in what form. We further summarize evaluation dimensions, benchmark gaps, and open challenges, and argue that future research should move from static, pointwise compression toward dynamic, governable, lifecycle-aware memory management.

1 Introduction

Long-context inference is constrained not only by context capacity but also by computation, storage, and the need to select useful information. This survey frames compression as lifecycle management across compute-active, compute-inactive, and intermediate memory states.

1 Introduction

  • Long inputs increase inference costs while repeated, obsolete, or irrelevant content can dilute critical evidence, making information management a finite-budget problem.The survey identifies selection, compression, and use of information as central alongside capacity.Source: §1 · pp. 1–2
  • The framework distinguishes compute-active memory from compute-inactive memory and includes intermediate states that can be restored without re-retrieval or re-encoding.The distinction depends on participation in computation at a given inference step, not storage location.Source: §1 · p. 2
  • Writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting are modeled as state transitions under a shared budgeted cost model.The model places these operations in a single cost account.Source: §1 · p. 2
  • The survey reviews methods across the information lifecycle and summarizes evaluation dimensions, benchmark gaps, and open challenges.Its scope includes input, representation, and KV-cache compression, external long-term memory, learned policies, and transition mechanisms.Source: §1 · p. 2

2 Background and Positioning

As context windows expand, the challenge shifts from how much information a model can receive to what should enter inference and at what cost. Prior surveys often separate related areas, while this survey compares them by computational participation, transitions, and a budgeted cost model.

2.1 From Context Extension to Context Management

  • Long-context management must balance growing computation and cache costs against evidence fidelity, temporal consistency, and future recoverability.Over-compression can break evidence chains; over-retention can increase latency, memory pressure, and noise.Source: §2.1 · p. 3

2.2 Existing Surveys and the Gap Addressed by This Survey

  • Prior surveys often separate current-inference compression and cache management from long-term memory, leaving transitions between internal states and external storage less analyzed.Existing unified memory views organize the field by cognitive analogy, functional role, or operation type.Source: §2.2 · pp. 3–4
  • This survey organizes methods by computational participation, distinguishes re-encoding retrieved text from recalling encoded states, and separates online, offline, and amortized costs.The distinction matters because retrieve-and-inject and recall fetch memory at different costs.Source: §2.2 · p. 4

3 Memory-Centric Framework for Long-Context Compression

The framework treats long-context compression as inference-time memory management: it allocates information across active, inactive, and intermediate states, and evaluates operations within a shared cost account.

3 Memory-Centric Framework for Long-Context Compression

  • Long-context compression is framed as allocating inference-time computation by deciding when information participates, in what form, and at what cost.The framework connects memory states, state-transition operators, a lifecycle, and a cost model.Source: §3 · p. 4

3.1 Memory States and Computational Participation

  • Compute-active memory participates in current inference, compute-inactive memory is maintained outside it, and encoded intermediate states can return through data movement and light correction.The distinction is based on participation in computation, not storage location.Source: §3.1 · pp. 4–5

3.2 State-Transition Operators

  • State-transition operators distinguish retrieve-and-inject, which re-prefills information from inactive memory, from recall, which mainly moves already encoded states.Compression is a representation change attached to operators; writing copies rather than removes its source, while eviction discards information.Source: §3.2 · p. 6

3.3 Information Lifecycle in Long-Context Systems

  • The lifecycle spans writing and organizing memories, retrieving and preparing them for inference, then writing back updates or forgetting outdated, conflicting, low-value, or sensitive content.Compression is continuous rather than one-time input pruning.Source: §3.3 · pp. 6–7

3.4 A Budgeted Cost Model

  • The budgeted cost model combines per-unit holding costs across active, intermediate, and inactive states with the costs of transitions performed for an inference.It frames compression as maximizing output utility under cost and active/intermediate-state budgets; reducing active cost can shift cost to transitions or offline maintenance.Source: §3.4 · pp. 7–8

3.5 Analytical Dimensions for Method Comparison

  • Methods are compared by memory state and operators, compression target, selection signal, system requirements, and evaluation criteria.Evaluation includes computational and storage cost, task utility, evidence fidelity, recoverability, and long-term consistency.Source: §3.5 · p. 8

4 Compressing Compute-Active Memory

Compute-active compression reduces the cost of information already on the inference path, spanning prompt admission, compact representations, cache management, architectural changes, and serving systems.

4.1 Input and Prompt Compression

  • Prompt compression admits context before inference using redundancy, token-budget, or query-relevance signals; outputs remain inspectable, but may lose source traceability or evidence needed for later reasoning.It works best when task goals are clear, redundancy is high, and evidence can be located from the query.Source: §4.1 · pp. 8, 10

4.2 Compressed Representations: Gist Tokens, Latent States, and Summary Memory

  • Compressed representations encode long contexts as denser states, avoiding repeated encoding but making preserved facts hard to inspect and transfer across models or closed APIs.Examples include learned gist tokens, segment-wise summary vectors, and compressed activations or bounded long-term memory.Source: §4.2 · pp. 10–11

4.3 KV Cache Compression

  • KV-cache methods preserve predicted future-use states through recency, attention, quantization, merging, or paging; reasoning models also compress cache states generated by long reasoning traces.Quest keeps the full cache resident and loads pages with the highest attention upper bounds for the current query.Source: §4.3 · p. 11

4.4 Architecture-level Selective Computation

  • Architectural methods embed memory selection or compaction in attention or alternative history-access mechanisms; adoption usually requires architecture or kernel changes and often training, ruling out closed-model interfaces.InfLLM is an exception: it is training-free and runs on a pretrained model at inference time.Source: §4.4 · pp. 11–12

4.5 Serving-level Cache and Prefix Management

  • Serving methods place, reuse, schedule, and release computed states across memory and requests, converting local savings into end-to-end gains whose size depends on workloads, hardware, frameworks, and cache hit rates.Evaluation should report throughput, memory utilization, hit rates, and tail latency, not only single-request speed.Source: §4.5 · p. 12

5 Compute-Inactive Memory Compression

Compute-inactive memory stores information outside the current computation path for later retrieval and reuse, so its management includes writing, organization, retrieval, maintenance, and learned policies.

5.1 Memory Writing

  • Writing selects interaction, task, or tool-feedback content for long-term memory; complete histories reduce omission but increase store size, retrieval noise, privacy risks, and stale or erroneous content.Written objects include factual, episodic, trajectory, reflective, and skill memories.Source: §5.1 · pp. 12–13

5.2 Memory Organization

  • Memory organization structures and indexes entries to reduce noise at later retrieval and injection, but richer structures complicate maintenance, while coarse or overly fine units impair relevance or contextual integrity.Systems use temporal streams, vector indexes, graphs, hierarchical stores, and linked notes, among other forms.Source: §5.2 · p. 13

5.3 Memory Retrieval and Activation Readiness

  • Retrieval selects memories for the query, task state, or goal and compresses the candidate space; retrieving too little risks forgetting or inconsistency, while retrieving too much adds noise and downstream cost.Retrieved memories remain candidates until selection, compression, and injection make them compute-active.Source: §5.3 · p. 13

5.4 Memory Update, Consolidation and Forgetting

  • Updating reconciles changing or conflicting information, consolidation increases memory density while risking lost detail, and forgetting removes or down-weights low-value, stale, erroneous, or sensitive content.Together these operations help maintain the store's signal-to-noise ratio over time.Source: §5.4 · p. 14

5.5 Learned Memory-Management Policies

  • Reinforcement learning can optimize memory-management decisions under a budget, but final-answer rewards do not directly constrain memory faithfulness, traceability, or privacy.Incorporating cost and reliability terms into the reward remains open.Source: §5.5 · p. 14

6 State Transitions: Retrieval, Injection, and Intermediate States

Section 6 treats information flow between memory states as an activation-control problem: retrieved content affects generation only after selection and injection, while intermediate states can return to computation with less re-encoding. It also examines agent-controlled compression and risks across the write-back loop.

6.1 From Retrieved Memory to Active Context

  • Retrieved content must be selected, compressed or structured, and injected; wholesale injection restores context costs, while aggressive filtering risks losing task-critical information.The needed information varies by task, including verifiable evidence, consistent user state, code dependencies, and tool results or action histories.Source: §6.1 · pp. 14–15

6.2 Compression and Injection Forms

  • Post-retrieval options range from filtering and summarization to structured facts, graph paths, latent representations, and KV states, with trade-offs in cost, interpretability, and detail retention.Text or structured facts favor traceability, latent representations favor extreme compression, and prefix or cache states favor multi-turn reuse.Source: §6.2 · p. 15

6.3 Intermediate States on the Activation Path: Offloading and Materialization

  • Offloading historical KV states enables recall of selected states during decoding without re-prefilling, but selection and recall remain sensitive to bandwidth, index construction, and precision.Methods differ in how they identify and fetch useful states; some move states to host memory, while others select which states to read.Source: §6.3 · pp. 15–16
  • Materialization precomputes external memory into loadable states, reducing later activation to loading and correction but creating model-version, storage, and inspection challenges.Cartridges distill synthetic corpus conversations into a smaller trainable KV cache targeted at an expected query distribution.Source: §6.3 · p. 16

6.4 Agent-Side Activation Control

  • Agent-side activation control compresses accumulated trajectories through learned guidelines, folded sub-trajectories, compact internal states, or bounded in-context memory.The agent policy becomes the decision-maker, while whether folded trajectories should be written back and how lost information might be recovered remain open questions.Source: §6.4 · p. 16

6.5 Feedback and Failure Modes

  • Because errors and poisoned memories can persist through retrieval and write-back, validation must cover writing, materialization, injection, and write-back.The loop can preserve erroneous outputs or reflections for future retrieval, and compression can amplify poisoned content.Source: §6.5 · p. 16

7 Evaluation, Benchmarks, and Metrics

The evaluation section argues that compression must be assessed through cost and task utility across the memory lifecycle, rather than by information removed alone. Existing benchmarks cover important but partial capabilities, leaving lifecycle-wide evaluation and cost accounting incomplete.

7 Evaluation, Benchmarks, and Metrics

  • Effective evaluation should pair cost–quality curves across context lengths and budgets with task utility, since shorter inputs or smaller caches can damage evidence, reasoning, user state, or consistency.Reported costs should include active holding, intermediate-state holding, transitions, and offline maintenance, with amortization made explicit.Source: §7 · p. 17, §7.1 · p. 17

7.1 Cost, Utility, and Long-term Reliability

  • Evaluation should measure evidence retention, faithfulness, reasoning consistency, hallucination, and task success, alongside long-term writing, retrieval, updating, consolidation, and forgetting reliability.Single-answer optimization can still accumulate erroneous reflections, stale preferences, or conflicting states.Source: §7.1 · p. 17

7.2 Benchmark Gaps and Evaluation Protocols

  • Benchmarks cover long-document understanding, extreme-length retrieval, and cross-session memory capabilities, with recent suites adding application tasks, semantic association, multi-fact reasoning, KV workflows, and stale-memory recognition.The cited examples include HELMET, NoLiMa, BABILong, SCBench, MemoryAgentBench, and STALE.Source: §7.2 · p. 17
  • Current benchmarks emphasize static inputs or isolated capabilities, and no benchmark jointly covers KV-level compression, memory updating, and systematic cost reporting.The proposed protocols separate comprehension, localization, retrieval, updating, forgetting, and agent execution while varying budgets and interaction rounds.Source: §7.2 · pp. 17–18, Table 4 · p. 18

8 Open Challenges and Future Directions

The open challenges concern making memory allocation adaptive across tiers, trustworthy across write-back and forgetting, and measurable over the full lifecycle. The survey also identifies reasoning traces, multimodal contexts, and parametric representations as emerging pressures on the memory-management problem.

8.1 Learned, Cross-Tier Budget Allocation

  • Fixed compression, retrieval, and cache policies overlook task-dependent needs and interactions, motivating joint allocation across input compression, KV retention or offloading, retrieval, and injection.The survey asks whether one learned policy can optimize these choices under its cost model; existing learned policies do not include explicit costs or KV- and serving-level decisions.Source: §8.1 · pp. 18–19

8.2 Trustworthy Write-back, Security, and Governable Forgetting

  • Write-back errors can persist as lasting bias, so memory governance requires evidence checking, provenance, conflict detection, versioning, user control, and tracking derived representations for correction or deletion.The survey identifies invalidating derived summaries, embeddings, graph nodes, materialized KV states, and compressed caches as an under-studied challenge.Source: §8.2 · p. 19

8.3 Lifecycle-Level Evaluation with Full Cost Accounting

  • Evaluation remains largely centered on single inputs, answers, or fixed ratios, while lifecycle evaluation should test whether memory stays usable, trustworthy, and governable over time.The survey notes that no benchmark combines lifecycle coverage with itemized costs, although existing benchmarks cover parts of the KV and memory-update lifecycle.Source: §8.3 · p. 19

8.4 Emerging Frontiers: Reasoning Traces, Multimodality, and the Parametric Boundary

  • Reasoning traces, multimodal contexts, and parametric representations complicate memory allocation through new active-memory sources, distinct redundancy structures, and blurred boundaries between context and weights.The survey asks when text, materialized KV states, or parameters are appropriate across query frequencies, corpus sizes, and update rates.Source: §8.4 · p. 20

9 Conclusion

The conclusion reframes long-context compression as managing information for inference-time computation, including what to retain, activate, update, or forget. It calls for dynamic, joint, and governable memory management evaluated across the full lifecycle.

9 Conclusion

  • The survey unifies prompt and cache compression, selective computation, serving, external memory, retrieval augmentation, agent memory, and learned policies as lifecycle mechanisms balancing storage cost, computation, and task utility.Its framework distinguishes compute-active and compute-inactive memory, intermediate activation-path states, and state transitions under a budgeted cost model.Source: §9 · p. 20
  • Future systems should allocate memory across tiers according to task and budget, prevent error accumulation through write-back, support verifiable forgetting, and report full lifecycle costs.The conclusion calls for moving beyond static, pointwise, modular compression toward dynamic, joint, governable memory management.Source: §9 · p. 20
Loading 2610.21165v1…