Source-linked AI summary
ATANT: An Evaluation Framework for AI Continuity
Samuel Sameer Tanguturi
TL;DR
Existing AI systems provide memory-related components but lack a formal way to establish genuine continuity across time. ATANT defines and evaluates continuity through required properties, checkpoints, and narrative tests, with the reference implementation reaching 96% at cumulative 250-story scale.
Problem
Existing components such as long-context windows, RAG pipelines, profile layers, and vector databases do not individually or jointly formally establish continuity as defined by the paper.
Method
ATANT defines continuity through 7 required properties and evaluates it with a model-independent 10-checkpoint methodology and narrative test corpus.
Results
96% cumulative-scale accuracy was achieved by the reference implementation, alongside a legacy-to-current progression from 58% to 100% in reported test settings.
Takeaways & Limitations
ATANT frames continuity as an architectural property addressable through deterministic engineering rather than probabilistic tuning.
Takeaways & Limitations
The evaluation checks keyword verification rather than reconstruction quality, so future versions should add reconstruction-quality metrics.
Abstract
from arXiv · showhide
We present ATANT (Automated Test for Acceptance of Narrative Truth), an open evaluation framework for measuring continuity in AI systems: the ability to persist, update, disambiguate, and reconstruct meaningful context across time. While the AI industry has produced memory components (RAG pipelines, vector databases, long context windows, profile layers), no published framework formally defines or measures whether these components produce genuine continuity. We define continuity as a system property with 7 required properties, introduce a 10-checkpoint evaluation methodology that operates without an LLM in the evaluation loop, and present a narrative test corpus of 250 stories comprising 1,835 verification questions across 6 life domains. We evaluate a reference implementation across 5 test suite iterations, progressing from 58% (legacy architecture) to 100% in isolated mode (250 stories) and 100% in 50-story cumulative mode, with 96% at 250-story cumulative scale. The cumulative result is the primary measure: when 250 distinct life narratives coexist in the same database, the system must retrieve the correct fact for the correct context without cross-contamination. ATANT is system-agnostic, model-independent, and designed as a sequenced methodology for building and validating continuity systems. The framework specification, example stories, and evaluation protocol are available at https://github.com/Kenotic-Labs/ATANT. The full 250-story corpus will be released incrementally.
1 Introduction
ATANT addresses the lack of a formal, testable framework for continuity beyond session-based context, retrieval, and memory components. It defines continuity requirements, evaluates them with narrative tests, and reports reference-implementation performance across sequenced compliance levels.
- Session-based AI systems preserve prompt context, conversation history, or retrieved notes, but this is insufficient for meaningful relationships over time.
- Existing components such as long context, RAG, profile layers, and vector databases do not individually or jointly produce continuity as defined here.ATANT distinguishes continuity from determining what persists, what changes, what remains relevant, and how context is reconstructed.
- The field lacks a formal definition, testable requirements, and a benchmark measuring continuity rather than retrieval accuracy alone.
- ATANT defines continuity as an architectural layer with 7 required properties and introduces a model-independent 10-checkpoint methodology without an LLM in the evaluation loop.
- The framework supplies 250 narrative stories, 1,835 questions across 6 life domains, and 4 sequenced compliance levels for progressive evaluation.
- 96% cumulative-scale accuracy marks the reference implementation’s active frontier across 5 test suite iterations.
2 Related Work
Prior systems address persistence and memory mechanisms, but ATANT positions continuity as a higher-order property requiring evaluation beyond isolated fact retrieval. It combines full-pipeline testing, naturalistic narratives, and cumulative disambiguation under memory load.
- MemGPT, Mem0, and A-MEM provide tiered, production-oriented, or self-organizing memory mechanisms for persistent AI systems.
- These systems can store and retrieve facts without formally defining or testing disambiguation, reconstruction, or temporal ordering as continuity requirements.
- CMA defines 6 behavioral properties for long-horizon agents but focuses on memory mechanisms and provides no evaluation corpus.
- The Narrative Continuity Test offers theoretical grounding for identity persistence across 5 dimensions but provides neither implementation nor test corpus.
- ATANT tests the full write-to-read pipeline with naturalistic multi-turn narratives and cumulative disambiguation, rather than retrieval alone or synthetic fact pairs.
- ATANT evaluates whether combined system components produce continuity, rather than replacing memory systems or retrieval benchmarks.
3 Defining Continuity
ATANT defines continuity as carrying forward relevant prior context, updating it as reality changes, and reconstructing it appropriately for the current situation. It distinguishes this system property from storage and similarity retrieval, then specifies 7 required properties.
- Continuity carries forward what still matters from prior interactions, updates it when reality changes, and reconstructs useful context for the current situation.
- Memory stores the past, whereas continuity keeps the right parts active and can reconstruct relationships, feelings, changes, and unresolved matters.
- Retrieval returns text similar to a query, while continuity reconstructs the current state, including changed, active, and superseded information.
- The framework defines 7 properties that any system claiming continuity must satisfy, derived empirically by testing narratives and identifying failures when properties are absent.
4 ATANT: Framework Design
ATANT defines continuity as a distinct architectural layer and evaluates it through deterministic, model-independent checkpoints that progress from isolated to cumulative and scale testing.
- ATANT evaluates the continuity layer rather than the intelligence layer, without including a model in the evaluation loop.
- The framework verifies both whether facts are stored correctly and whether answers are retrieved and reconstructed correctly.
- Testing proceeds through isolated, stress, cumulative, and scale phases, with each phase targeting a harder property.
- ATANT defines 10 checkpoints spanning write-path verification, read-path verification, and cross-cutting concerns.
- CP8, the Final Answer checkpoint, is definitive, while the other checkpoints diagnose failure locations.
- ATANT defines 4 compliance levels with Gold at 100%, Silver at 95–99%, and Bronze at 90–94%.
5 Narrative Test Corpus
The ATANT corpus models ongoing human life through realistic, adversarial multi-turn narratives across six domains, with structured expected memories and verification questions.
- ATANT stories simulate realistic multi-turn conversations spanning hours, days, or weeks across personal, private, emotional, and ongoing life contexts.
- The corpus includes multi-fact utterances, pronoun chains, temporal updates, negation, emotional overlays, and ambiguous predicates as adversarial patterns.
- The six domains are Career, Relationships, Health, Learning, Daily Life, and Life Events.
- Each story contains metadata, timestamped conversation batches, expected memory stores, and verification questions with expected keywords.
- A verification question passes when all expected keywords appear in the retrieved answer, using case-insensitive and substring-permissive matching.
6 Experiments and Results
The reference implementation improved through architectural redesign, reaching perfect isolated performance but retaining a 4% gap at 250-story cumulative scale because similarly named predicates compete across narratives.
- The NURA Memory Pipeline was evaluated across 5 test suite iterations, with all evaluations independent of LLMs.
- The legacy pipeline reached a ceiling at 58% and regressed under tuning pressure, while the redesigned architecture reached 100% on 250 stories.The redesign reached 100% within 6 days, and the paper characterizes the improvement as architectural rather than tuning-based.
- Cumulative mode tests whether the system retrieves the correct fact for the correct context without cross-contamination when multiple life narratives coexist.
- Suite 1.2 failed on 12 stories and 15 questions involving niche predicates, and the fix expanded the Predicate Lexicon architecturally.
- At 250-story cumulative scale, 74 questions failed because similarly named predicates from different narratives competed, leaving a 4% gap.Disambiguation requires context, entity, and trace convergence.
- CP4 Type Tagging scored 51.4% on exotic domain-specific objects, but these diagnostic failures did not affect CP8 accuracy.
7 Discussion
ATANT’s results support treating continuity as an architectural property evaluated across model-independent checkpoints and cumulative memory loads. The discussion also identifies limitations in reconstruction measurement, corpus diversity, system coverage, and language scope.
- 7.1 Continuity Is an Architecture Problem: 58% →100% demonstrates that continuity cannot be achieved through scoring optimization alone.The progression is attributed to grammar-first classification, deterministic trace convergence, and structural matching—architectural decisions rather than hyperparameter changes.
- 7.2 Cumulative Evaluation: Cumulative mode tests disambiguation under memory load, where 250 life narratives share storage and semantic similarity can conflate distinct events.This distinguishes retrieval performance from continuity evaluation.
- 7.3 Model Agnosticism as Future-Proofing: ATANT evaluates continuity without any model in the loop, defining properties and checkpoints independently of current AI architectures.The framework is intended to remain applicable as the intelligence layer changes.
- 7.4 Limitations: CP8 verifies expected keywords in retrieved answers rather than measuring reconstruction quality, so coherent answers are not directly assessed.Future versions should add reconstruction quality metrics.
- 7.4 Limitations: All 250 stories were written by one author, limiting linguistic diversity and cultural representation.The corpus therefore does not fully represent varied language use or cultural contexts.
- 7.4 Limitations: Only one system has been evaluated against ATANT, leaving the framework’s value dependent on testing by independent systems.The authors invite other teams to run ATANT and publish results.
- 7.4 Limitations: The corpus is English only and does not test multilingual continuity.Multilingual narratives are identified as a future extension.
8 Conclusion
The conclusion presents ATANT as a framework that formalizes and evaluates AI continuity through defined properties, checkpoints, a narrative corpus, and compliance levels. It positions continuity as an architecture problem and notes planned extensions to the framework.
- 8 Conclusion: ATANT provides 7 continuity properties, 10 evaluation checkpoints, a corpus of 250 stories and 1,835 questions, and 4 compliance levels.These components define the framework and its sequenced validation structure.
- 8 Conclusion: The reference implementation demonstrates that continuity is an architecture problem addressable through deterministic engineering rather than probabilistic tuning.This is the conclusion’s central claim about how the evaluated system achieves continuity.
- 8 Conclusion: Future versions will add reconstruction quality metrics, multilingual narratives, proactive behavior testing, and community-contributed stories.Version 1.0 defines the framework’s foundation while these capabilities remain planned extensions.
- 8 Conclusion: The framework specification and evaluation protocol are available at https://github.com/Kenotic-Labs/ATANT.The repository provides access to the published materials.