Source-linked AI summary
Will My Assistant Remember My Allergy? What Personal LLM Assistants Forget When Conversation Memory Is Compressed
Lichen Zhu, Yueqian Lin, Yiheng Wang, Yudong Liu, Hai "Helen" Li, Yiran Chen
TL;DR
Personal assistants need to retain facts across conversations, but on-device memory is constrained and standard benchmarks let eviction policies see the question in advance. The paper evaluates cache reuse with the question hidden until after compression; on PA-Bench, dormant-fact recall falls to 0.00–0.01, versus 0.97 with full memory.
Problem
On-device personal assistants must retain useful conversation facts despite scarce memory, but query-visible benchmarks do not reflect cache reuse before a future question exists.
Source: §1 · p. 2, §3.1 · p. 2Method
The paper compares query-hidden cache reuse with query-visible evaluation, testing whether policies retain facts when compression precedes the recall question.
Source: §3.1 · p. 2, §4 · p. 3, §5.1 · p. 3Results
On PA-Bench, query-hidden dormant-fact recall is 0.00–0.01, against 0.97 with full memory.
Source: §8 · p. 6Takeaways & Limitations
Safety-critical facts should live in an auditable episodic memory alongside the compressed cache, and the assistant should ask rather than invent when it forgets.
Source: §8 · p. 6PA-Bench is templated, and the study’s single-mask, block-granular budgeting and one-shot selection limit its scope; real conversations and repeated eviction remain to be tested.
Source: §7 · p. 5
Abstract
from arXiv · showhide
Personal LLM assistants (health companions, elder-care agents, accessibility aides) are judged by what they remember about a person: a medication or an allergy mentioned in passing and needed days later. Privacy pushes them on-device, where a month of conversation can outgrow the model's own weights, so an eviction policy must decide what the cache forgets. Benchmarks report that eviction keeps such facts at a 20% budget, but they compress a prompt that already contains the user's future question, foresight no cache-reusing assistant has. Hide the question until after compression and the advantage vanishes: on PA-Bench, 100 assistant conversations we construct, an allergy mentioned in passing survives to the question that needs it 0--1% of the time, against 97% with full memory. The cause is the budget, not the scorer: none of the training-free policies we evaluate ranks the fact high enough, and the budget that would keep it is too large to bother compressing. A compressed cache is an inference-reuse mechanism, not a persistence layer: safety-critical facts need an auditable episodic store alongside it, and an interface that asks rather than invents.
1 Introduction
Personal assistants depend on memory over time, but on-device constraints make conversation memory costly and require eviction policies. The paper argues that query-visible benchmarks overstate what cache-reusing assistants can retain, and introduces a query-hidden evaluation of that setting.
1 Introduction
- A passing allergy disclosure can determine whether later health advice is safe; when the measured assistant forgets, it asks rather than invents in 99% of failures.That failure is recoverable if the interface surfaces it.Source: §1 · p. 2
- Privacy pushes assistants on-device, where the KV cache holding past turns grows linearly with conversation and can rival model weights within weeks, requiring eviction.Eviction decides what the assistant forgets [13].Source: §1 · p. 2
- Prior benchmarks report dormant-fact retention at a 20% cache budget, but measure query-visible batch compression rather than causal, stateful cache reuse [6] [10] [13].The evidence therefore does not answer which policy an assistant should ship.Source: §1 · p. 2
- Standard benchmarks compress a prompt that already contains the recall question; hiding that future question reverses their reported policy ranking.A cache-reusing assistant must compress before the later question exists.Source: §1 · p. 2
- The study targets bounded, incrementally compressed KV caches; its critique does not apply unchanged to stateless systems that re-prefill the raw transcript.Query-hidden evaluation isolates stateful reuse, while one-shot selection is an optimistic abstraction of incremental eviction.Source: §1 · p. 2
- The paper contributes a query-hidden protocol and PA-Bench, where no evaluated policy retains a fact block at 20%; it assigns durable safety-critical facts to an episodic store, not cache compression.It also finds no evaluated training-free policy ranks the dormant fact high enough to survive an affordable budget.Source: §1 · p. 2
2 Related Work
Related work spans human-centered personal agents and multiple approaches to memory. This paper focuses on what a bounded, reused KV cache can retain without knowing the future query.
2 Related Work
- Eviction policies range from query-aggregate scorers such as H2O, DefensiveKV [6], and EpiCache [10] to recency methods such as StreamingLLM [22] and LRU.Other approaches use a recent window, a predicted query, or query-agnostic compression [13] [21] [1] [9].Source: §2 · p. 2
- Long-range recall benchmarks include LoCoMo [16], LongMemEval [20], LongBench [2], and RULER [7], but most leak the query into compression.The paper frames its question as what cache retention can be trusted to carry without query foresight.Source: §2 · p. 2
- Healthcare conversational agents and on-device personal LLM agents are studied in human-centered systems [12] [14] [15] [8].These systems include health companions and agents built for efficiency and privacy.Source: §2 · p. 2
- Modern agent stacks distinguish the KV cache, which accelerates inference over active context, from episodic or structured stores that hold durable facts.Examples include MemGPT [18], Mem0 [4], and A-MEM [23].Source: §2 · p. 2
3 Two Flaws in the Standard Evaluation
The paper identifies query leakage and error-prone stateful reimplementations as flaws in standard evaluation. Hiding the query changes what policies can retain, while corrected implementations alter dormant-fact recall.
3.1 The benchmark leaks the future question
- Standard multi-turn benchmarks compress the conversation with its recall question already visible, letting query-specific scorers keep the queried block using future information.Cache-reusing assistants must discard tokens before that question arrives.Source: §3.1 · p. 2
3.2 Reimplementation breaks the baselines
- Correcting SnapKV’s scorer raises its mean recall, pooled over both planted facts in the audit harness, from 0.12 to 0.94.The faulty key-similarity proxy ignored rotary position encoding and degraded across long position gaps, unlike the attention-based specification [13].Source: §3.2 · p. 3
- Restoring H2O’s cumulative attention makes it the only tested policy that recalls anything under a query-hidden protocol.The mistaken implementation used a single appended probe token instead of accumulating attention over all query positions.Source: §3.2 · p. 3
- Published SnapKV and H2O results are not challenged for single-prefill, query-visible use; moving them to cache reuse is easy to get wrong, motivating a shared query-hidden harness.The corrected code reproduces SnapKV’s 0.975 query-visible result [13].Source: §3.2 · p. 3
4 A Leak-Free Evaluation Protocol
The protocol isolates whether a scorer can preserve facts without seeing the later question. It compares query-visible compression with query-hidden cache reuse while matching the other harness conditions.
4 A Leak-Free Evaluation Protocol
- A dormant fact offers a query-blind scorer only the attention it drew when mentioned; cumulative lifetime attention is the only tested signal with a chance to retain it.Current-position attention from a probe or a window past the fact scores it near zero by construction.Source: §4 · p. 3
- The evaluation reimplements SnapKV with windowed query-key attention and H2O with cumulative attention, alongside StreamingLLM, LRU, a probe, and an INT4 demotion variant.Policies share block granularity and matched budgets, counting attention sinks and recency window within the budget.Source: §4 · p. 3
- Query-visible compression scores a prefill containing the recall question; query-hidden compression occurs before the question exists and reuses the surviving cache without rescoring.The question is then prefilled onto retained blocks at their original positions.Source: §4 · p. 3
- Both protocols hold the model, budgets, block granularity, and decoding constant, while one-shot compression at question time optimistically abstracts incremental eviction.Thus the scorer’s available information is the protocol difference, and one-shot selection may overstate what repeated eviction preserves.Source: §4 · p. 3
5 Results and Analysis
Hiding the future question reverses the apparent advantage of query-specific eviction: on assistant conversations, no evaluated compressed policy retains the passing safety-critical fact at a 20% budget. The fact returns only when the budget reaches its rank, while query-aware scoring or more cache capacity erodes the point of compression.
5.1 Hide the question and the best scorer keeps nothing
- With the question hidden, query-specific scorers fall to 0.00 on the needle, while H2O reaches 0.17; on PA-Bench, even H2O falls to 0.00.The full cache scores 0.92 in the query-hidden setting, indicating that eviction, rather than model capability, is the bottleneck.Source: §5.1 · p. 3, Table 1 · p. 4, Table 2 · p. 4
- On established long-context benchmarks, corrected eviction policies are statistically indistinguishable from the full cache when evidence remains salient as the question arrives.For example, LongBench QA SnapKV F1 is 0.156 against probe’s 0.151 (bootstrap p=0.63); LoCoMo is saturated at its 8K cap.Source: §5.1 · p. 3, §5 · p. 3
5.2 No evaluated policy keeps the allergy in assistant conversations
- On PA-Bench, query-hidden H2O retains 0.00 of the fact’s blocks, versus 0.86 when the query is visible.The benchmark uses 100 thirty-round conversations with safety-critical facts disclosed in passing and queried at the end; H2O ranks fact blocks 16–59 of 161, beyond the approximately seven score-driven slots at 20%.Source: §5.2 · p. 4, Table 3 · p. 4
5.3 The allergy returns only when compression stops paying
- At 20%, approximately seven score-driven slots miss H2O’s fact ranks of 16–59; recall returns as the budget expands to reach those ranks.Sinks and recency reserve approximately 25 blocks; the passage reports approximately 39 slots at 40%, 71 at 60%, and 103 at 80%.Source: §5.3 · p. 4, Table 4 · p. 5
- At an affordable budget, a query-aware SnapKV scorer reaches 0.975, while a generated candidate question reaches 0.09 and INT4 demotion raises recall only from 0.00 to 0.01.Recovering dormant facts without the query requires keeping enough cache that compression no longer pays; H2O is also the most expensive policy to score.Source: §5.3 · pp. 4–5
6 Implications for Assistant Builders
Compressed KV caches support inference reuse, not durable memory, so assistant builders must pair them with an episodic store and repair-oriented interactions. That store shifts the bottleneck to recognizing facts worth saving, while personalization remains a proposed way to improve recall.
6 Implications for Assistant Builders
- 0.51 of cases saw the episodic store recover the allergy, against 0.00 for any compressed cache.The store’s recall equals its extraction coverage, so it does not back-stop facts it fails to recognize as worth storing.Source: §6 · p. 5
- At a nominal 20% budget, sinks and recency leave ∼7 of 32 kept blocks for scoring—a 4% scoring cache.Choosing a policy by the nominal budget can therefore silently under-retain.Source: §6 · p. 5
- At 56 KB/token, conversation memory exceeds the model’s 3.8 GB INT4 weights within a month, and compacting yields up to 2.4× faster decoding.The caption describes Table 5’s measurements for Qwen2.5-7B and notes that decoding is memory-bound.Source: Table 5 · p. 5, §6 · p. 5
- At 20% query-hidden, no evaluated policy retained the health fact, while query-aware scorers kept it in 86% of cases under the leaky protocol.The retention curve also tracks disclosure persistence; cache eviction is not a deletion guarantee.Source: §6 · p. 5
- On PA-Bench, 99% (196/198) of failures were repair requests, and none confabulated a different drug or contact.A repair-oriented interface can surface uncertainty and let the user re-supply or correct the missing fact.Source: §6 · p. 5
- No evaluated training-free policy is both effective without foresight and deployable; user-specific prediction from routines and calendars is proposed as a promising lever.The authors recommend that evaluators hide the recall query during compression and report full-memory performance at every budget.Source: §6 · p. 5
7 Future Work
The future-work agenda calls for testing the results on real conversations and complete on-device memory pipelines, and for measuring repeated eviction. It also identifies untested image and sensor settings and possible future-aware approaches.
7 Future Work
- PA-Bench’s templated conversations should be checked against span-annotated real conversations, and evaluation should cover full on-device pipelines trading retrieval accuracy, latency, and energy against cache reuse.The study also leaves repeated per-turn eviction to future measurement because its one-shot selection is an optimistic abstraction.Source: §7 · p. 5
- The single-mask, block-granular budget scopes the results to training-free policies and may understate per-head methods; finer granularity may recover part of the gap.This does not change what a query-blind scorer can see.Source: §7 · p. 5
- Figure 2 reports H2O recall of 0.43 at 40% and 0.80 at 60%; its naive scoring implementation OOMs past 4K tokens.The caption says an efficient kernel would accumulate column-sums instead.Source: Figure 2 · p. 6
- Whether image and sensor context obeys the same query-blind limits remains untested, despite its larger KV footprints.Future-aware policies may close the gap at the cost of training, profiling, or a reconstruction pass.Source: §7 · pp. 5–6
8 Conclusions
With the question hidden until after compression, dormant-fact recall at deployable budgets is near zero, while full memory retains high recall. The paper concludes that compressed caches should serve inference reuse, with safety-critical facts kept in auditable episodic memory.
8 Conclusions
- 0.00–0.01 query-hidden recall replaces 0.83–0.87 query-visible recall on PA-Bench, against 0.97 with full memory.These figures describe dormant safety-critical facts at deployable budgets.Source: §8 · p. 6
- Dormant-fact recall is impractical under the studied compression budgets, not impossible.Source: §8 · p. 6
- The paper recommends treating compressed caches as inference-reuse mechanisms and storing safety-critical facts in auditable episodic memory alongside them.Whether the assistant remembers an allergy should be measured rather than assumed from a leaky benchmark.Source: §8 · p. 6