PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning

Yehya Farhat, Michael Desmond, Anastasios Kyrillidis

arXiv:2610.02687v1cs.AIcs.LG

TL;DR

Fixed-weight models do not naturally carry lessons between interactions, while flat external memories expose growing context to each query. GraphMemory frames memory updates as context optimization and retrieves bounded, query-relevant subgraphs; across evaluated settings, it retains competitive accuracy with substantially fewer construction tokens.

  • Problem

    Flat shared memories expose increasingly large and heterogeneous contexts to each query, raising token use and context-window pressure as experience accumulates.

    Source: §1 · p. 2

  • Method

    GraphMemory stores reusable knowledge as graph nodes with weighted relations and retrieves a bounded, query-relevant subgraph for each query.

    Source: §2.3 · p. 4

  • Results

    Across three dataset–model settings, GraphMemory stays within 4.0 percentage points of ACE with Claude Haiku 4.5, exceeds ACE by 1.5 points on FORMULA with Qwen3.8-27B, and cuts training-token usage 81–85%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under bounded block size and retrieval count, the memory context supplied per query remains O(1) as examples accumulate.

    Source: §1 · p. 2, §2.3 · p. 4

    Evaluation remains preliminary across two related tasks, two models, and one primary baseline, and retrieval overhead makes individual test queries slower.

    Source: §5 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed in enterprise, scientific, and medical applications, where agents must incorporate domain-specific knowledge and adapt from experience. Context engineering offers a practical alternative to weight updates by improving model behavior through instructions, strategies, and evidence supplied at inference time. However, adapting context online typically requires a costly trial-and-error process, while queries are often processed independently, preventing useful experience from carrying forward. Memory systems address this limitation by retaining information across interactions, but approaches that continually append information to a shared context face increasing token costs, context-window limits, and performance degradation as the context expands. We introduce a unified formulation of context optimization and show that an agent memory system update can be interpreted as an optimization update procedure over the model's context. This perspective attempts to provide a principled framework for studying memory design and its efficiency. We then propose GraphMemory, a lightweight graph-based memory that accumulates, refines, organizes, and connects reusable strategies. For each query, GraphMemory retrieves only the relevant subgraph, enabling online context adaptation without exposing the model to the entire memory. Under bounded retrieval, the amount of retrieved memory remains constant as the number of processed examples grows. Experiments show that GraphMemory achieves competitive downstream performance while using approximately 81-85% fewer memory-construction tokens than our baselines.

1 Introduction

Deployed models often cannot carry experience forward through their weights, while shared external memories can become costly as they grow. GraphMemory organizes reusable strategies as a graph and retrieves only relevant knowledge, reporting competitive performance with lower token use.

1 Introduction

  • Fixed-weight models do not naturally improve from one interaction to the next, while adaptation methods requiring weights and training infrastructure constrain continual deployment and transfer.External natural-language memory offers a weight-free alternative, but shared flat memories can increase per-query token use and approach context limits.Source: §1 · pp. 1–2
  • Across three dataset–model settings, GraphMemory stays within 4.0 percentage points of ACE with Claude Haiku 4.5, beats ACE by 1.5 points on FORMULA with Qwen3.8-27B, and cuts training-token usage 81–85%.It also reduces test-token usage by 79–92% and active training time by 21–34%.Source: §1 · p. 2
  • GraphMemory stores reusable ACE-style knowledge points as nodes, connects related points with weighted edges, and retrieves relevant local subgraphs rather than prompting with the full memory.This preserves a natural-language memory while making relationships among lessons explicit.Source: §1 · p. 2

2 Context and Memory Formalization

The section frames memory as context optimization: lessons can be generated locally from experience, but their composition and retrieval determine what reaches the model. Append-only shared contexts grow with experience, whereas bounded retrieval can keep the supplied memory context bounded.

2.1 Offline Context Optimization

  • Context optimization chooses textual context c to minimize empirical task risk over an offline dataset, although discrete tokens and black-box access prevent ordinary first-order optimization.The context may contain instructions, demonstrations, knowledge, or strategies.Source: §2.1 · p. 2
  • Prompt-optimization methods search a budget-limited candidate pool, yielding a task-level context that remains fixed on the test set.The methods differ in how they construct and search the candidate pool.Source: §2.1 · p. 3
  • Feedback-driven methods propose context edits through a black-box oracle, an update structurally analogous to gradient descent but not a literal gradient or guaranteed improvement.The edit may add, delete, replace, or otherwise revise context.Source: §2.1 · p. 3

2.2 From Local Context Updates to Memory

  • Memory systems simplify update generation by distilling a focused strategy, correction, or failure mode from an individual experience into a context block.The collected blocks form an external memory bank that serves as the memory system's evolving optimization state.Source: §2.2 · p. 3
  • The local block-risk surrogate is not claimed to optimize the global objective: a block may overfit its source example, so a separate composition or retrieval mechanism must select lessons for new inputs.The local formulation models memory updates and makes each oracle task more focused; shared-memory systems integrate lessons into one task-level context.Source: §2.2 · p. 3
  • With at most one bounded-length block per example, append-only memory context grows as O(N); measured ACE prompt-token growth follows this trend over the evaluated horizon.Repeatedly supplying the full memory makes cumulative memory-context exposure grow quadratically across sequential updates.Source: §2.2 · pp. 3–4, §2.3 · p. 4

2.3 Memory with Bounded Retrieval

  • Bounded retrieval separates accumulated memory from exposed context by selecting query-dependent blocks; with bounded block size and retrieval count, the supplied context remains bounded as examples accumulate.GraphMemory instantiates this interface with a graph, while the proposition bounds Generator input—not storage or retrieval cost.Source: §2.3 · p. 4

3 GraphMemory

GraphMemory modifies ACE's read and write interfaces: it retrieves a bounded neighborhood for each query and lets curation add or revise graph relations as well as knowledge nodes. The graph supplies connected lessons to the model without loading every full memory entry.

3 GraphMemory

  • On FORMULA with Qwen3.8-27B, the plotted comparison tracks GraphMemory and ACE prompt-token usage across successive training checkpoints; compare their growth trajectories.The caption identifies the systems, task, model, and checkpoint progression but states no outcome.Source: Figure 1 · p. 5, §3 · p. 5
  • A compact index supplies candidate entry nodes, then deterministic breadth-first traversal expands neighbors under fixed hop, node, and token budgets before loading their contents for generation.The retriever makes the initial semantic selection; graph traversal brings in connected knowledge.Source: §3 · p. 4, Figure 2 · p. 5
  • Helpful and harmful feedback updates confidence-weighted edge evidence, while the Curator can add or revise nodes and edges locally; neutral feedback leaves edge counters unchanged.The traversal follows edges by the mean of their resulting Beta posterior, although the growing index means the implementation is not strictly O(1) end to end.Source: §3 · p. 5

4 Experiments

Across two financial reasoning tasks, GraphMemory is compared with ACE on two model backbones for adaptation performance, token use, and efficiency. It uses substantially fewer tokens while maintaining competitive accuracy, with lower active training time but higher per-query inference time.

Experimental setup

  • The evaluation compares GraphMemory with ACE on FINER and FORMULA using Claude Haiku 4.5 and Qwen3.8-27B, with each system trained for one epoch.ACE memory is capped at 80,000 tokens for Claude but not for Qwen3.8-27B.Source: §4 · pp. 5–6

Adaptation performance

  • GraphMemory uses 81–85% fewer memory-construction tokens and 78.6%–92.2% fewer test tokens than ACE while maintaining competitive accuracy across three dataset–model configurations.The comparison is summarized in Table 1.Source: Table 1 · p. 6, §4 · p. 6
  • On FORMULA with Qwen3.8-27B, GraphMemory slightly outperforms ACE using only 19% of ACE’s training-token budget; on Claude/FORMULA, ACE leads by 4.0 points using 6.6× as many training tokens.On FINER, GraphMemory trails ACE by 2.94 points.Source: §4 · p. 6
  • GraphMemory improves more slowly and varies more initially, but continues improving as its graph grows, while ACE improves rapidly before plateauing or declining near its 80,000-token cap.The trajectories suggest a trade-off between early sample efficiency and late-stage stability.Source: §4 · p. 6, Figure 3 · p. 6

Wall-clock efficiency

  • GraphMemory reduces active training time by 21.0% on FORMULA and 34.3% on FINER, but takes approximately 1.8× as long per test example as ACE.The inference overhead comes from retrieval before each prediction.Source: Table 2 · p. 7, §4 · p. 7

Token and monetary efficiency

  • On Claude Haiku 4.5/FORMULA, GraphMemory uses approximately 20M versus 131M training tokens, an 84.7% reduction, and lowers estimated construction cost from $157.20 to $24.00.Across Qwen/FORMULA and Claude/FINER, estimated costs fall from $26.25 to $5.00 and from $392.40 to $68.40, respectively.Source: §4 · p. 7

5 Conclusion, Limitations, and Future Work

The paper frames memory construction as iterative optimization over textual context and proposes GraphMemory to retrieve bounded, query-relevant context. Results are competitive with ACE while using fewer tokens, but the evaluation remains preliminary and limited in scope.

5 Conclusion, Limitations, and Future Work

  • Across two financial tasks and two LLM backbones, GraphMemory remains competitive with ACE while reducing memory-construction tokens by 81–85%, test-time tokens by 79–92%, and estimated API cost similarly.Learning trajectories suggest slower initial adaptation but greater stability and continued improvement.Source: §5 · p. 7
  • GraphMemory organizes ACE-style procedural knowledge as a graph, learns relations from confidence-weighted helpful and harmful feedback, and retrieves bounded, query-relevant context.The formulation contrasts constant retrieved content per query with linear memory-context growth under append-only prompting.Source: §5 · p. 7
  • The results are preliminary, covering two related tasks, two models, and one primary baseline; repeated runs and broader evaluations are needed.Longer continual-learning streams and cross-model transfer remain future tests.Source: §5 · p. 7

A Additional Formalization

The appendix formalizes context optimization and memory updates, motivates local context blocks as a practical surrogate, and compares the scaling of append-only memory with bounded retrieval.

A.1 Textual Optimization Oracles

  • MIPROv2 searches candidate instructions and demonstrations per module, while GEPA adaptively grows a candidate archive using execution traces and feedback.Both are abstracted as candidate-pool searches whose resulting context is fixed during test-time inference.Source: §A.1 · p. 10
  • OPRO, TextGrad, and TRACE instantiate iterative textual context updates, but a proposed update direction requires evaluation or acceptance to verify objective reduction.The gradient-descent analogy refers to update structure, not differentiability or a convergence guarantee.Source: §A.1 · p. 10

A.2 Why Use a Local Context-Block Surrogate?

  • A per-example block can overfit by encoding its source answer, so the context-block surrogate is not equivalent to the global objective.Its purpose is to model memory updates and make each oracle generation task more local.Source: §A.2 · p. 10
  • The local context-block risk decomposes memory updates into focused lessons from individual experiences, rather than directly optimizing the global context objective.A separate composer or retriever must expose reusable blocks on relevant new inputs, and final risk must be validated on composed contexts.Source: §A.2 · p. 10

A.3 Memory-Context Scaling

  • Θ(N) final context length and O(N^2) cumulative memory-context exposure follow for append-only updates under the stated block-size assumptions.The token bound concerns memory supplied to the model, not all system prompt and completion tokens.Source: §A.3 · p. 11
  • With blocks of length at most B and at most K retrieved per query, retrieval-based memory has O(1) per-query exposure and O(N) cumulative exposure.These bounds exclude the stored graph, condensed index, and context consumed internally by the composer or retriever.Source: §A.3 · p. 11

A.4 GraphMemory as Structured Retrieval

  • GraphMemory stores context blocks as graph nodes and relationships as edges, then retrieves and serializes a query-dependent subset.Supplying all nodes is the full-memory special case; bounded retrieval instead limits exposure to a query-dependent set.Source: §A.4 · p. 11

B.1 Directed Evidence-Weighted Memory Graph

GraphMemory represents reusable context blocks and their directed relationships in a graph, using confidence-weighted evidence to rank transitions during retrieval.

B.1 Directed Evidence-Weighted Memory Graph

  • Each directed edge stores separate nonnegative evidence counters for helpful and harmful transitions, with new edges initialized from a uniform Beta(1, 1) prior.The posterior mean ranks edges during retrieval, and opposite directions have distinct counters and weights.Source: §B.1 · pp. 11–12
  • The Reflector reports an ordered influence trace, whose adjacent loaded-node pairs define candidate directed transitions for evidence updates.Only identifiers among nodes loaded for that example are accepted; duplicate pairs and invalid feedback are discarded.Source: §B.1 · p. 12
  • Confidence controls the fractional evidence contributed by each observation, while neutral feedback or zero confidence leaves counters unchanged.The same directedness allows transitions in opposite directions to accumulate different retrieval weights.Source: §B.1 · p. 12
  • Retrieval weight estimates how often the Reflector judged an observed transition helpful under the current policy, not causal influence or calibrated accuracy gain.The estimate is policy-dependent, and ranking by posterior mean does not explicitly reward uncertainty.Source: §B.1 · p. 12

B.2 Retrieval and Traversal

GraphMemory retrieves a bounded query-specific subgraph, then updates its graph only after generating the current example's prediction.

B.2 Retrieval and Traversal

  • Retrieval selects up to two index sections and six entry nodes, then traverses outgoing edges by posterior-mean weight for at most two hops, 12 nodes, and 2,500 serialized tokens.Breadth-first order is preserved, with edge scores ordering neighbors at each expansion point.Source: §B.2 · pp. 12–13
  • If the graph is empty or no valid entry is selected, retrieval returns no memory block rather than falling back to the full memory.Retrieval traces record selected and loaded nodes, edges, token count, depth, and timing.Source: §B.2 · p. 13

B.3 Retrieve–Generate–Reflect–Curate Execution Order

  • For each training example, GraphMemory retrieves from the graph built from earlier examples before generating its prediction.The Reflector and Curator act only after prediction, so information derived from the current label becomes available to later examples.Source: §B.3 · p. 13

B.4 Model and Evaluation Configuration

The experiments compare GraphMemory and ACE using matched model configurations within each dataset–backbone comparison and select checkpoints by validation accuracy.

B.4 Model and Evaluation Configuration

  • ACE and GraphMemory share the same backbone and model configuration within each comparison, including the specified Qwen runtime choices and Claude default request configuration.For Qwen3.8-27B, the settings include 4-bit quantization, SDPA, and disabled thinking mode.Source: §B.4 · p. 13
  • Each system trains for one epoch, evaluates validation accuracy every 100 updates, and uses the highest-accuracy checkpoint for held-out testing.Formula uses 500/300/200 train/validation/test examples; FiNER uses 1,000/500/441.Source: §B.4 · pp. 13–14

B.5 Token and Latency Accounting

The constant retrieved-content bound applies only to serialized node content supplied to the Generator, not total system usage. The complete index grows with indexed nodes, and GraphMemory’s additional retrieval call can increase per-query latency despite lower token use.

B.5 Token and Latency Accounting

  • The constant bound covers only serialized node content supplied to the Generator, not the complete system input.Retriever index, retrieved content, fixed instructions and query tokens, and other component inputs and outputs contribute to total usage.Source: §B.5 · p. 14
  • Because the Retriever reads the complete index, its input grows with the number of indexed nodes.Source: §B.5 · p. 14
  • GraphMemory can use fewer tokens while having higher per-query latency because its wall-clock measurements include an additional retrieval call.Source: §B.5 · p. 14
Loading 2610.02687v1…