PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing

Jiahe Geng, Jinpeng Wang, Kun Yuan

arXiv:2609.04915v1cs.AI

TL;DR

Long-horizon LLM agents need memory designs that preserve answer quality under tight prompt budgets rather than relying on full-context prompting. RSM-full addresses this with max-member merging and grouped context packing, reaching a strong compact-memory quality–token operating point on AMA-Bench and reproducing the pattern on RealMem.

  • Problem

    The central problem is identifying which memory design offers the best quality–token trade-off in the compact-memory regime as full-context prompting becomes costly with interaction length.

  • Method

    RSM-full is an online clustered-memory pipeline combining a cosine-threshold max-member merge write rule with an atom-aware grouped context packer.

  • Results

    Across AMA-Bench and RealMem, RSM-full establishes a strong compact-memory operating point, with AMA-Bench reaching 83% of Full-Context quality at 32% of the token cost at a 4k prompt budget.

  • Takeaways & Limitations

    Under tight prompt budgets, compact-memory performance depends mainly on how streaming memories are merged and retrieved content is assembled, especially around 2k–5k prompt tokens.

  • Takeaways & Limitations

    The positive end-to-end claims use one embedder and one primary judge stack, while raw LoCoMo and LongMemEval are boundary regimes where the compact-memory advantage weakens or disappears.

Abstract

from arXiv · show

Many long-horizon LLM deployments face tight prompt budgets: latency, cost, and context limits make full-context prompting impractical as interaction length grows. The key question is then not raw recall alone, but which memory design gives the best quality--token trade-off in the compact-memory regime. We present \textbf{RSM-full}, an online clustered-memory pipeline designed for a strong quality--token Pareto point. RSM-full combines two design choices: a cosine-gated \emph{max-member merge} write rule and an atom-aware grouped context packer. On AMA-Bench, our primary compact-memory benchmark, it reaches $83%$ of Full-Context quality at $32%$ of the token cost at a $4$k budget; under four-seed averaging it beats the closest streaming-clustered baseline (Online K-Means) by $+3.5$--$6.0$,pp ($p{<}.001$) across the whole ${\sim}2.6$k--${\sim}5$k regime. Three-seed ablations show most of this gain comes from the merge rule ($+5.7$,pp over Online K-Means and matched-$τ$ DP-means) and the grouped packer ($+5.0$,pp over flat concatenation). The pattern reproduces on RealMem, an independent long-horizon persona-memory benchmark: RSM-full improves on Budget-RAG ($+0.69$,pp, $p{=}.006$), is on par with BM25-RAG (paired $Δ{=}{+}0.27$,pp, $p{=}.47$; we do \emph{not} claim BM25 equivalence in the equivalence-test sense), and significantly outperforms Streaming-Proto ($+2.97$,pp) and the closest reproduced 2025 agentic-memory baseline A-MEM ($+1.65$,pp, $p{<}.001$). Across benchmarks the message is consistent: under tight budgets, compact-memory performance is driven mainly by how streaming memories are merged and how retrieved content is assembled. Overall, RSM-full is most useful when answeroughly $2k$--$5k$ prompt tokens, where itdefines a strong compact-memory Pareto point; higher-token baselines remain stronger outside this regime.

1 Introduction

RSM-full targets the quality–token trade-off in compact-memory prompting with an online clustered-memory pipeline combining max-member merging and grouped context packing. Across AMA-Bench and RealMem, it establishes a strong compact-memory operating point, while retrieval evidence and benefits weaken in specified boundary regimes.

  • Core approach: RSM-full combines a cosine-threshold max-member merge write rule with an atom-aware grouped context packer.The pipeline is evaluated on AMA-Bench and independently on RealMem.
  • AMA-Bench result: 83% of Full-Context quality at 32% of the token cost at a 4k prompt budget on AMA-Bench.Under matched four-seed averaging, it beats Online K-Means by +3.5–6.0 pp across the ∼2.6k–∼5k regime.
  • Scope: The v1 retrieval rule is conditional: its contrast is null on BGE-normalized AMA-Bench once clustering and packing are fixed, but it improves 2/4 LoCoMo-Plus grouped Cognitive relation families.Raw LoCoMo and LongMemEval are boundary regimes where the compact-memory advantage weakens or disappears.
  • Mechanism account: +5.7 pp over Online K-Means and matched-τ DP-means and +5.02 ± 1.00 pp over flat concatenation in matched AMA-Bench ablations.These ablations attribute most of the improvement to the merge rule and grouped packer.
  • RealMem replication: RSM-full improves on Budget-RAG, matches BM25-RAG within noise, and outperforms Streaming-Proto and A-MEM on RealMem under matched evaluation.RealMem is an independent long-horizon persona-memory benchmark.

2 Method

RSM-full’s compact-memory design centers on a permissive cosine-gated max-member write rule and atom-aware grouped assembly. The method retains higher-rank storage extensions, but the primary AMA-Bench quality advantage is attributed to clustering and packing.

  • Storage extensions: Higher-rank storage and quantised variants are retained as deployment extensions, while the primary BGE AMA results are explained by clustering and packing.The extensions include stored bases, coefficient summaries, Grassmann merge, adaptive rank, int8, and PQ.
  • Cosine-gated max-member merge: RSM incrementally routes each incoming trajectory chunk to an existing atom or allocates a new atom using a cosine-gated write rule.The rule uses the running ℓ2-normalised mean of each atom.
  • Cosine-gated max-member merge: A new chunk merges when either the centroid or any existing member exceeds threshold τ; otherwise, RSM allocates a new atom.This max_mem check is more permissive than pure centroid thresholding and is tested against Online K-Means and matched-τ DP-means.
  • Cosine-gated max-member merge: τ is calibrated per embedding space as approximately the 0.70 quantile of pairwise cosines from at most 50 unlabelled validation examples, with O(Md) write cost per chunk.M denotes the live atom count.
  • Atom-aware grouped context assembly: RSM retrieves top-K atoms, ranks members within each atom, and packs selected chunks by atom with headers and temporal ordering rather than as a flat list.This grouped assembly is the second core component.
  • Atom-aware grouped context assembly: 5.02 ± 1.00 percentage points are lost when atom-aware grouping is replaced with flat concatenation in a matched three-seed AMA ablation.Clustering and retrieval are held fixed in this comparison.

3 Experimental Setup

The experiments use complementary benchmarks and matched baselines to test compact-memory quality–token trade-offs, component ablations, and transfer to long-horizon persona memory. Hyperparameters are selected on disjoint validation splits, with benchmark-specific agents, judges, and metrics.

  • Benchmarks: AMA-Bench contains 208 episodes and is the primary compact-memory benchmark for the quality–token operating point and merge-and-packing ablations.RealMem tests transfer to long-horizon persona memory under matched evaluation.
  • Baselines: The AMA-Bench comparison includes Full-Context, retrieval systems, streaming prototypes, agentic-memory baselines, Online K-Means, and matched-τ online DP-means.RealMem includes BM25-RAG, Full-RAG, Budget-RAG, Streaming-Proto, RSM variants, and reproduced A-MEM.
  • Selection protocol: All RSM hyperparameters are selected on held-out validation splits strictly disjoint from reported test sets.The threshold calibration uses approximately the 0.70 quantile of pairwise cosines from at most 50 unlabelled validation examples.
  • Evaluation protocol: AMA-Bench uses GPT-4o-mini as agent and a uniform llmmelon GPT-5.41 rejudge over all 2,496 QAs per method.RealMem uses GPT-4o-mini with llmmelon GPT-5.4 and pools three stream-permutation seeds.
  • Evaluation protocol: Evaluation metrics vary by benchmark: LLM-as-judge scores AMA-Bench, RealMem, and LoCoMo-Plus, while raw LoCoMo and LongMemEval use reference token-F1.ToolBench uses GPT-4o as both agent and judge; MMLU and LLaMA hidden-state experiments use Recall@1.

4 Main Results

RSM-full establishes a favorable compact-memory quality–token operating point on AMA-Bench and reproduces its advantage on RealMem. Matched analyses attribute most of the gain to max-member merging and atom-aware grouped packing, while results outside the target regime remain conditional or weaker.

  • AMA-Bench operating point: 83% of Full-Context quality is achieved at 32% of the token cost at a matched ∼4k AMA-Bench budget.Higher-token references remain stronger in absolute quality but lie outside the compact-memory frontier.
  • AMA-Bench operating point: +3.7/+3.5/+6.0 pp separates RSM-full from Online K-Means at ∼2.6k/∼4k/∼5k under four-seed averaging.The advantage persists across the tested compact-memory budget range.
  • Why it works: +5.7 pp over Online K-Means and +5.5 pp over matched-τ DP-means identify the max-member merge rule as the principal clustering gain.The retrieval-rule contrast is null when clustering and packing are held fixed (A−D, p=0.40).
  • Why it works: +5.02 ± 1.00 pp results from replacing flat concatenation with atom-aware grouped packing in a matched three-seed ablation.The comparison holds clustering and retrieval fixed, making prompt assembly a substantive component of the result.
  • Why it works: +4.90 pp remains for RSM-full over K-Means when both use the atom-aware packer, supporting a clustering contribution beyond assembly.This bridge check uses the same packer to reduce the assembler confound.
  • Scope and boundary conditions: At N=32k, all methods reach the No-Memory ceiling, so the long-horizon check supports compression behavior rather than end-to-end quality.Removing the max-member fallback increases atom count by +8.84% / +10.56% at null paired quality cost.
  • RealMem replication: RealMem reproduces the compact-memory pattern: RSM-full improves on Budget-RAG, matches BM25-RAG within noise, and beats Streaming-Proto and A-MEM.The positive end-to-end claims are scoped to AMA-Bench, RealMem, and grouped LoCoMo-Plus slices; raw LoCoMo and LongMemEval are boundary conditions.

5 Discussion

The evidence hierarchy places AMA-Bench at the center of the compact-memory claim, with RealMem providing independent replication and ablations attributing gains mainly to merging and grouped packing. Retrieval-rule benefits remain conditional, while higher-token methods can remain stronger outside the target regime.

  • AMA-Bench establishes the main operating-point result, RealMem reproduces it independently, and matched ablations attribute the gain mainly to max-member merging and grouped packing.
  • The v1 retrieval rule is null on BGE-normalized AMA-Bench after clustering and packing are fixed, but activates on 2 of 4 LoCoMo-Plus grouped relation families.
  • Entries report quality averages and mean prompt tokens, with paired McNemar significance markers defined as p<.05, p<.01, p<.001, and NS.
  • RSM-cent-retr retains RSM-full’s clusters and atom-aware packer while replacing retrieval with centroid cosine based on the first basis direction.
  • ToolBench and MMLU indicate usefulness under matched-byte storage and anti-forgetting tests, but higher-token methods remain stronger above 12k tokens.

6 Limitations

The evaluation’s strongest end-to-end claims rely on one BGE embedding stack and one primary judge stack, while several benchmark settings provide only conditional or harness-limited evidence.

  • Positive end-to-end claims are anchored by AMA-Bench and RealMem under a single BGE embedding stack.
  • End-to-end experiments use one embedder and one primary judge stack, leaving cross-embedder replication for future work.
  • LoCoMo-Plus grouped Cognitive supplies only conditional supporting evidence, while raw LoCoMo and LongMemEval are boundary regimes where compact-memory advantage weakens or disappears.
  • RSM does not support selective forgetting, and keyword-heavy settings can favor lexical or hybrid retrieval.

Societal Impact

Reducing per-agent memory footprint does not remove the need for safeguards against leakage, demographic clustering, and judge-amplified errors.

  • RSM reduces per-agent memory footprint but still requires safeguards against cross-session leakage, demographic-bucket clustering, and judge-amplified errors.

7 Conclusion

RSM-full defines a strong quality–token Pareto point under tight budgets, especially when streams contain reusable latent structure. The paper also identifies deployment variants and supplementary storage machinery, while outlining future work on selective forgetting, replication, and validation.

  • 7 Conclusion: RSM-full is most relevant when answer quality must be maintained under roughly 2–5k prompt tokens.
  • 7 Conclusion: Future work includes selective forgetting, cross-embedder replication, and broader multi-seed validation of appendix-level checks.
  • Variant definitions: RSM-full is the algorithmic instance, RSM-full-int8 is the storage-compressed deployment variant, and RSM-centroid is a centroid-only control.
  • Benchmark notes: AMA-Bench reports atom-level and episode-level proxies because each episode contains only ∼2.3 atoms on average.
  • Benchmark notes: MMLU uses a sequential-stream protocol and a different control baseline.
  • Supplementary controls: The matched-qid LoCoMo-Plus control reports flat ∆=+2.5 pp (p=1.00) and grouped ∆=+10.0 pp (p=0.22), with the grouped effect directional rather than conclusive at n=40.
  • Storage extensions: Higher-rank storage is supplementary; on MMLU sequential, k∈{2, 4, 8} are equivalent in quality, with RSM-full-int8 at k=2 the lowest-byte operating point.

B.2 Algorithm E: Subspace-Projected Per-Chunk Scorer (Negative Result)

Algorithm E replaces RSM-full’s ambient-cosine per-chunk scorer with a subspace-projected inner product while keeping the write gate, atom filter, and grouped packer unchanged. On the full AMA-Bench comparison, this change does not improve results, supporting clustering and packing—not deeper basis use—as the main BGE gains.

  • Method: Algorithm E replaces ambient cosine with a subspace-projected inner product while preserving RSM’s write gate, atom filter, and atom-aware packer.The two systems differ only in the per-chunk scoring formula.
  • Method: Every j≥2 summand contributes only when both query and chunk align with a non-principal atom direction, potentially exposing signal from otherwise unused basis columns.The modification was intended to test whether higher-rank storage could help retrieval scoring.
  • Protocol: The comparison uses the full 208-episode AMA-Bench with n=2,496 paired QA, rank-8 storage, and 10,000-resample paired bootstrap evaluation.Both methods use the same chunking, budget, packer, agent, and judge settings.
  • Result: The subspace-projected scorer does not outperform standard RSM-full, reinforcing clustering and packing as the main sources of the BGE gains.The full comparison is reported as negative, with no positive domain-level result reaching p<.05.
  • Latency: The deployed top-1 scorer remains O(Md), whereas full-basis scoring is an upper-bound variant with O(Mkd) cost and exceeds Full-RAG latency at M≥200.Precomputing basis columns makes the optimized implementation 10–20× faster than the naive version.

D AMA-Bench: Official GPT-4o Judge vs. GPT-5.4 Re-Judge

The compact-memory ranking is unchanged when AMA-Bench is evaluated with the official GPT-4o judge versus a uniform GPT-5.4 re-judge, although absolute Full-Context scores vary. Reliability checks indicate that the GPT-5.4 ordering is stable across episode splits and reruns.

  • Judge comparison: The compact-memory ranking is identical under the GPT-4o and GPT-5.4 judges at all three tested budgets.RSM-int8 leads at ≤4k, while RSM-full leads at ∼5k; Full-Context absolute scores differ across judges.
  • Judge comparison: Judge-model sensitivity affects absolute Full-Context scores but not the within-budget compact-memory ordering.The ranking pattern is therefore more stable than the absolute score scale.
  • Reliability: The GPT-5.4 reliability check splits the 208 episodes into odd and even halves and preserves the ordering RSM-full > Streaming-Proto > Budget-RAG on both halves.The half-sample margin is of the same order as the full-set margin.
  • Reliability: Independent GPT-5.4 reruns of RSM-full correlate at Pearson r=0.876 across 208 episodes, with run-level means of 0.318 and 0.320.These results are consistent with the reported 3× rerun standard deviation of 0.003.

E Reproducibility

The reproducibility evidence spans benchmark artifacts, matched-byte hidden-state experiments, robustness probes, and multi-seed comparisons. It supports strong online compact-memory performance while delimiting claims to the tested streaming settings and noting several boundary or reproduction limitations.

  • Artifacts: The release includes benchmark outputs, multi-seed sweeps, bootstrap files, robustness probes, and supporting references for AMA-Bench, RealMem, LoCoMo, ToolBench, and MMLU.The listed artifacts cover the paper’s primary comparisons and auxiliary experiments.
  • Matched-byte storage: RSM-full retains 99.9% Recall@1 at 812 KB, achieving 31.5× compression versus Full-RAG’s 25.6 MB and beating same-byte Budget-RAG by +11.0 pp.The claim is confined to the online streaming-clustered comparison set; offline RAG-KMeans is effectively tied.
  • Matched-byte storage: Across k∈{2, 4, 8}, RSM-full reaches 82.4% Recall@1 with full 10/10 subject coverage, while increasing k above 2 adds storage without improving quality.The k=2 int8 variant preserves quality at the lowest reported byte point of 88.5 KB.
  • Robustness: Under additive noise, RSM-full retains 101–105% of its σ=0 quality while Streaming-Proto and Budget-RAG degrade by 2–4 pp.Its head-to-head advantage widens from +0.5 pp at σ=0 to +2.9 pp at σ=2.0 before stabilizing.
  • Seed stability: Among streaming compressors, K-Means and Oja show 11–16% between-seed variation at realistic budgets, whereas RSM-full stays within 1–2%.The comparison is explicitly confined to streaming-clustered methods and does not claim parity with retrieval-only systems.
  • Adversarial robustness: At K=15 adversarial injection, RSM-full retains 88.9% of baseline quality and leads Streaming-Proto by +5.21 pp, but at K=30 all compact-memory methods converge near 0.32.The saturation regime corresponds to foreign chunks exceeding roughly 60% of the memory pool.

G RSM-cent-merge ablation: detailed data

The RSM-cent-merge ablation isolates the max-member fallback gate’s effect on atom-count compression in stitched AMA-Long trajectories. Figure 4 illustrates sublinear atom growth and atom inflation without supporting a long-horizon quality claim, whose interpretation is limited by a benchmark ceiling.

  • Setup: The ablation compares RSM-full with a write-side RSM-cent-merge variant at N=16k and N=32k, changing only the merge rule.Both arms share atom-aware packing, τ=0.85, retrieval k=5, the same stream, and the same QA pipeline.
  • Scope: The write-side ablation measures atom-count compression rather than horizon-scaling quality.The trajectories are stitched from AMA-Bench episodes, and the authors explicitly restrict the evidence to the merge mechanism.
  • Figure 4: Figure 4 plots RSM-full atom growth from N=1k to 32k against a raw-chunk linear reference and overlays RSM-cent-merge points at N=16k and N=32k.The intended comparison is sublinear growth versus the linear reference and atom inflation from removing the fallback gate.
  • Quality caveat: At N=32k, RSM-full accuracy is 0.179 versus No-Memory’s 0.176, with paired Δ=+0.33 pp and p=.79 NS.The null quality gap reflects the benchmark ceiling rather than a method-level horizon conclusion.
  • Reproduction caveat: MemoryBank and MemGPT reproductions on RealMem scored at or below the No-Memory floor under the stated harness.The authors treat these outcomes as harness-level reproduction limitations rather than method-level evidence.
Loading 2609.04915v1…