Source-linked AI summary
The Compaction Cliff in Long-Running AI Agent Memory
Saber Zerhoudi, Jelena Mitrovic, Michael Granitzer
TL;DR
Long-running agents can lose safety rules when uniform compaction treats them like ordinary context, producing the Compaction Cliff. Knowledge Triage classifies knowledge by type and applies dedicated operators for compaction, decomposition, and retrieval. Across public corpora and downstream benchmarks, it preserves safety better than production and uniform baselines, while its evaluation remains bounded by public-data coverage and classifier reliability.
Problem
Uniform context management can paraphrase or drop safety rules alongside ordinary knowledge, with Sonnet 4.6 retaining 53% after one compaction round and 10% after five.
Method
Knowledge Triage classifies each knowledge item by type and applies TypeCompact, TypeDecompose, and TypeRetrieve with type-specific retention policies.
Results
Across five public corpora, typed operators preserve safety constraints better than uniform strategies, including 2–4× higher compaction recall, 0% versus 93% decomposition locality violations, and 100% versus 73% retrieval recall@50.
Takeaways & Limitations
Per-type retention provides a unified way to protect safety-critical knowledge across compaction, decomposition, retrieval, and downstream agent behavior.
Takeaways & Limitations
The evaluation uses public GitHub artifacts, classifier labels with lower agreement away from the safety-critical split, and multi-round rollout tests on only two of four LLM families.
Abstract
from arXiv · showhide
A safety rule and an episodic log compete for the same tokens in an AI agent's context. When the budget overflows, both are summarized at the same rate; only the rule needs exact wording to remain enforceable. On 20 production agent configurations, Claude Code's /compact prompt on Sonnet 4.6 preserves 53\% of safety rules after one compaction round and 10\% after five. We name this the Compaction Cliff. We address it with Knowledge Triage, a framework that classifies each line of an agent's knowledge base by type and routes each type through its own retention policy. Three deterministic operators implement this triage across the three context-management operations: TypeCompact rewrites items in place under per-type fidelity, TypeDecompose partitions a topic too large to compact safely, replicating in-scope safety rules across partitions, and TypeRetrieve fetches items from external storage with in-scope rules pinned ahead of relevance. On five public corpora, TypeCompact preserves 2--4$\times$ more safety rules than the strongest single-shot LLM compactor at every ratio, with 96\% recall over five rounds. TypeDecompose reaches 0\% locality violations against 93\% under uniform partitioning. TypeRetrieve reaches 100\% recall@50 against 73\% for the best single-shot LLM retriever. On three downstream behavioral benchmarks, we outperform the production Sonnet compactor on medical compliance (paired McNemar $p < 10^{-8}$ on preservation, $N = 200$), the full-policy and hierarchical baselines on retail task pass rate ($p < 0.01$, $N = 115$), and the hierarchical compaction on the airline domain ($p = 0.024$). We release AgentArtifactCorpus (396{,}934 agent configurations from 54{,}628 public GitHub repositories), the classifier, and the reference implementation.
1 Introduction
Production agents treat heterogeneous knowledge uniformly during context management, allowing safety rules to disappear under compaction. The paper names this degradation the Compaction Cliff and proposes Knowledge Triage as a type-aware alternative.
- Problem: Compaction, decomposition, and retrieval share a finite budget over heterogeneous knowledge, but production strategies do not condition retention on item type.A single utility can therefore treat safety constraints and background beliefs as equally disposable.
- Problem: Different knowledge types tolerate different losses: safety rules require exact preservation, procedures require behavioral equivalence, and debugging traces can be summarized.The paper argues that one compression policy cannot safely serve all three.
- Approach: Knowledge Triage classifies each item once and routes categories through type-specific retention policies across compaction, decomposition, and retrieval.The framework is implemented by TypeCompact, TypeDecompose, and TypeRetrieve.
- Contributions: The paper releases the dataset, classifiers, and reference implementation alongside its framework.
2 Related Work
Prior agent-memory research provides separate structures and retrieval or compression strategies, but does not combine the three context-management operations with per-type safety guarantees. Knowledge Triage extends typed knowledge into a unified context-management layer with exact preservation for constraints.
- Research gap: Prior work evaluates compaction, decomposition, and retrieval separately and lacks a formal per-type safety guarantee spanning all three.
- Typed memory: MaRS combines typed memory with structural safety properties but uses one submodular utility, so constraints can be lost at the same rate as background beliefs.
- Knowledge Triage: Knowledge Triage assigns every working-set item one of five types and gives each type its own retention rule across all three operations.The framework is presented as complementary to knowledge-graph memory and inference-time hierarchy enforcement.
- Compression and retrieval: Existing prompt compressors use uniform fidelity targets, while relevance-based retrieval methods do not condition ranking on constraint membership.The paper instantiates composite-source rate-distortion ideas with type-specific fidelity and exact preservation for constraints.
3 Knowledge Triage Framework
Knowledge Triage models agent knowledge by type, topic, and scope, then applies type-specific fidelity and safety rules through classification, compaction, decomposition, and retrieval operators.
- Typed model: Knowledge Triage routes five operational knowledge types through separate retention policies for compaction, decomposition, and retrieval.The types were derived from 396,934 agent knowledge artifacts, and Table 2 summarizes their distortion tolerance.
- Typed knowledge model: A typed knowledge base includes items, type assignments, a topic tree, leaf mappings, and constraint scopes, enabling decomposition and retrieval to preserve topical relationships.Global constraints can apply to every leaf topic even when stored under one local topic.
- Typed knowledge model: Five categories are the empirically coarsest partition supporting the per-operator safety arguments, with no added guarantees from finer subdivisions.
- Per-type fidelity: Each type receives its own distortion measure: constraints require zero distortion, procedures require behavior-preserving rewrites, and episodic items allow gist-level summarization.
- Classification: The only learned component is the classifier, so a missed constraint enters a soft retention lane where it may be paraphrased or dropped.The framework also weights confidence by source authority when estimating constraint risk.
- Classification: SafetyMargin classifies an item as a constraint when removing it could make some action unsafe, using a domain-conditioned action set and a 0.5 threshold.It is the default classifier, while cascades and surface heuristics trade safety recall for cost.
- Operators: TypeCompact preserves constraints under a feasible budget, TypeDecompose replicates each scoped constraint into every relevant partition, and TypeRetrieve pins in-scope constraints ahead of relevance.Uniform strategies violate these requirements when constraints fall into summarization windows, across partition boundaries, or below retrieval thresholds.
- Operator composition: The operators compose by attempting compaction first and falling through to decomposition and retrieval when the safe compaction budget is insufficient.
4 Empirical Validation
Across compaction, decomposition, and retrieval experiments, Knowledge Triage preserves safety-relevant information better than type-blind strategies, with classifier quality and typed routing determining deployment performance.
- Validation: 97% of agent items were covered by the five-type model, and human agreement was 0.92 on binary preserved-versus-lost decisions.The coverage set included 2,000 AAC annotations and 500 LongMemEval turns; annotator consensus agreed with automated decisions on 79% of cells.
- Compaction: 0.53, 0.39, and 0.24 were the best single-round recalls for type-blind strategies at 50%, 25%, and 10% compression.Five rounds reduced recall from 0.53 to 0.10 across every LLM family and structural baseline.
- Compaction: 0.96 constraint recall after five TypeCompact rounds, versus 0.10 for type-blind /compact.TypeCompact returned 1.00 / 0.95 / 0.80 recall at 50 / 25 / 10% compression and stabilized at 0.96 from the second round onward.
- Decomposition: 0% locality violations were achieved by TypeDecompose, compared with 13% for the strongest topic-aligned baseline.TypeDecompose incurred 14.5% mean token overhead; the topic-aligned baseline still produced violations in 40% of configurations.
- Retrieval: 100% in-scope recall@50 was achieved by TypeRetrieve across retrievers, while authority weighting retained recall@50 at 1.00 with a smaller pinned footprint.TypeRetrieve used zero LLM tokens per query, compared with 5,776–6,741 for single-shot LLM retrievers.
- Classifier cost and coverage: 0.93 constraint recall was reached by the best single-shot LLM classifier, while SafetyMargin held worst-case recall at 0.90 across phrasing forms.The AAC results show cheap variants at 0.27–0.77 recall and single-shot LLMs at 0.73–0.93; Opus 4.7 reached 0.73 because its trace overflowed the output format.
5 Discussion and Limitations
Knowledge Triage extends type-dependent retention across agent memory operations, while its guarantees depend on classifier recall and remain scoped to storage-layer behavior. Evaluation also has dataset, annotation, replication-cost, and rollout boundaries.
- Discussion: Knowledge Triage routes constraints through per-type retention policies across compaction, decomposition, and retrieval.A classifier is placed upstream of all three operations, with flagged constraints sent through an exact-preservation path.
- Guarantee conditions: Under a feasible budget, the operators preserve every constraint the classifier flags.The guarantee rests entirely on the classifier’s recall.
- Guarantee conditions: At SafetyMargin recall 0.93, the residual miss rate is 0.07; tighter bounds require higher-recall classification or an inference-time check.The passage attributes the residual gap to nine SafetyMed items.
- Limitations: The guarantee covers storage but not alignment training, chain-of-thought scaffolding, episodic consolidation, or inter-agent communication.Items whose constraint status depends on session context require an online classifier.
- Limitations: AgentArtifactCorpus comes from public GitHub, classifier distributions may differ in closed enterprises, five-class annotation agreement is κ=0.45, and TypeDecompose overhead ranges from 0% median to 219% worst case.The multi-round rollout covered two of four LLM families, and MaRS-FL is a reimplementation because MaRS lacks a public implementation.
6 Conclusion
Knowledge Triage classifies agent knowledge and applies type-specific retention across compaction, decomposition, and retrieval. Across public corpora and downstream benchmarks, it preserves safety constraints better than uniform strategies and improves or matches baseline task performance.
- Framework: Knowledge Triage assigns every knowledge-base item one of five types and routes each type through its own policy across three operations.The operators cover compaction, decomposition, and retrieval.
- Results: 2–4× higher constraint recall, 0% versus 93% locality violations, and 100% versus 73% recall@50 were reported for the typed operators against uniform strategies.These results span compaction, decomposition, and retrieval on five public corpora.
- Downstream behavior: On SafetyMed, TypeCompact led the production Sonnet compactor by 14.5 points on constraint preservation, with p<10^-8 and N=200.It also outperformed the Sonnet compactor on pass rate.
- Downstream behavior: On retail, TypeCompact outperformed the full-policy and hierarchical baselines on pass rate; on airline, it beat hierarchical truncation and tied the full-policy ceiling.The reported retail comparisons had p=0.003 and p=0.005; airline had p=0.024 against hierarchical truncation.
- Implication: Safety preservation at the storage layer is presented as a deployment requirement for production agents handling medical, legal, and customer-account tasks.The motivation concerns bounded-context systems already used in these settings.