PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

Sebastian-Antonio Toma, Jean de Kernier, Mike Ply, Oliviu Matei

arXiv:2610.22209v1cs.CLcs.AIcs.LG

TL;DR

Response-based memory benchmarks conflate memory correctness with retrieval and generation quality. FEB-30 generates documents from known fact trajectories to enable direct inspection of memory states. Intrinsic and extrinsic evaluation diverge by 16 percentage points on average, and FEB-30 exposes limits in the evaluated system’s operation classification.

  • Problem

    Agent-memory benchmarks typically use downstream question-answering accuracy as a proxy for memory quality, although this conflates memory correctness with retrieval and generation.

    Source: §1 · p. 1

  • Method

    FEB-30 generates documents from known fact-evolution trajectories, providing deterministic ground truth for direct memory-state evaluation.

    Source: §1 · p. 1, §4.2 · p. 3

  • Results

    Intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation reveals that the system correctly classifies memory operations only 20% of the time, a dimension response-based evaluation cannot measure.

    Source: §8 · p. 7

    FEB-30 covers factual evolution across multi-author artifacts, not episodic or procedural memory or single-turn conversational recall, and its 30 trajectories limit fine-grained statistical comparisons.

    Source: §7 · p. 7

Abstract

from arXiv · show

Agent memory benchmarks evaluate whether an agent's responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy for memory quality. We show this evaluation paradigm is incomplete: it conflates memory correctness with retrieval and generation quality, producing both false positives where correct responses mask broken memory and false negatives where incorrect responses blame correct memory for retrieval failures. We introduce FEB-30, a benchmark of 30 fact-evolution trajectories spanning 6 diagnostic categories and 3 difficulty levels, constructed via paradigm inversion: documents are generated from known fact trajectories, yielding deterministic ground truth without human annotation. Each trajectory defines how a fact should evolve across multi-author artifacts (documents, emails, conversations), enabling direct inspection of the memory store after each ingestion step. Applying FEB-30 to a representative memory system, we demonstrate that intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds. Diagnostic decomposition reveals that the system correctly classifies memory operations (create, update, reinforce, supersede) only 20% of the time, a failure entirely invisible to extrinsic evaluation, and that failure modes differ systematically by category and difficulty. We release the benchmark, document generation pipeline, and scoring framework to support reproducible evaluation of agent memory systems.

1 Introduction

Response-based benchmarks conflate memory correctness with retrieval and generation, so correct answers can mask faulty memory and failed retrieval can obscure correct memory. The paper proposes direct memory-state evaluation and introduces FEB-30 to diagnose these failures.

1 Introduction

  • Existing benchmarks evaluate memory through downstream answers, conflating memory, retrieval, and generation quality.A correct response cannot confirm that memory is correct, and a failed response cannot identify which component failed.Source: §1 · p. 1
  • Strong generation can mask incorrect or incomplete memory, while retrieval failure can make correctly stored facts appear wrong.These false positives and false negatives make response accuracy an unreliable proxy for memory quality.Source: §1 · p. 1
  • The paper proposes intrinsic evaluation: directly comparing memory-store contents with known ground truth at each stage of fact evolution.It complements end-to-end evaluation by exposing the memory component’s internal state.Source: §1 · p. 1
  • FEB-30 provides 30 fact-evolution trajectories with deterministic ground truth across six diagnostic categories.Its paradigm-inversion construction generates documents from known fact trajectories.Source: §1 · p. 1
  • The scoring framework measures intrinsic claim coverage, update detection, and false-update rate alongside extrinsic QA accuracy.Direct inspection supports diagnostic decomposition of failures by pipeline stage.Source: §1 · p. 2
  • Intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds.The divergence varies systematically by category.Source: §1 · p. 2

2 Related Work

Prior memory benchmarks generally assess memory through agent responses, while this paper extends ground-truth corpus generation to temporal fact evolution and direct memory inspection.

2 Related Work

  • Existing agent-memory benchmarks grade responses to queries and do not directly inspect the memory store.MEME includes controlled state trajectories but still evaluates through the response layer.Source: §2 · p. 2
  • Conflict benchmarks test reasoning about conflicts in retrieved context, rather than whether memory resolves conflicts during ingestion.The paper distinguishes this from memory-consolidation evaluation.Source: §2 · p. 2
  • FEB-30 extends paradigm inversion from static RAG ground truth to multidocument sequences whose ground truth is a trajectory of evolving states.The related approach, RIKER, generates evaluation corpora from known ground truth to avoid expensive, subjective annotation of real documents.Source: §2 · p. 2
  • The paper positions direct memory inspection as a complementary evaluation approach amid disagreement about whether it should be used.A concurrent survey documents the gap across 15 benchmarks and argues for complementary intrinsic evaluation; a practitioner framework calls direct inspection an anti-pattern.Source: §2 · p. 2

3 Benchmark Design: FEB-30

FEB-30 organizes fact evolution into diagnostic categories and difficulty levels, with trajectories specifying expected memory states and actions across documents.

3 Benchmark Design: FEB-30

  • FEB-30 comprises 30 fact-evolution trajectories organized by diagnostic category and difficulty level.The name analogizes the benchmark’s crossing of the threshold to intrinsic evaluation to the nonexistent date February 30.Source: §3 · p. 2

3.1 Diagnostic Categories

  • Its six categories test compensation, retrieval bottlenecks, operation classification, synthesis, temporal correctness, and multi-author attribution.C1 and C2 target intrinsic–extrinsic divergence; C3–C6 test dimensions where intrinsic evaluation provides richer signal.Source: §3.1 · pp. 2–3

3.2 Difficulty Levels

  • Difficulty increases from near-verbatim target claims at L1, through contextual extraction at L2, to inference amid substantial unrelated content at L3.The difficulty levels vary the extraction challenge and surrounding noise.Source: §3.2 · p. 3

3.3 Trajectory Schema

  • Each trajectory records expected memory states as atomic claims, new document claims, and expected actions for each state transition.Separating new per-state document claims from cumulative expected memory contents tests persistence across ingestions.Source: §3.3 · p. 3

3.4 Distribution

  • The benchmark has 30 trajectories, split evenly across three difficulty levels, spanning six technical domains and comprising 147 documents.The set includes 127 trajectory-state documents and 20 noise documents; the construction methodology is described as domain-agnostic.Source: §3.4 · p. 3

4 Document Generation Pipeline

The pipeline generates documents from known fact trajectories, using model rotation, difficulty-dependent styles, and independent verification to produce benchmark artifacts.

4 Document Generation Pipeline

  • Paradigm inversion generates documents from known fact trajectories, making the trajectories themselves the ground truth.This avoids annotating ground truth on real documents, which the paper describes as expensive and subjective.Source: §4 · p. 3

4.1 Multi-Model Rotation

  • Six models across five providers are rotated by state to simulate different authors, and no model generates more than 30% of documents.The extraction model uses a different LLM family from the generators to avoid stylistic bias that inflates scores.Source: §4.1 · p. 3

4.2 Style-Grounded Generation

  • L1/L2 documents mimic reference styles, while L3 injects claims into unrelated human-written documents that are approximately 90% authentic.L1 uses 200–400 words without noise; L2 uses 400–800 words with 2–3 unrelated claims; L3 uses 600–1000 words with 4–5 noise claims.Source: §4.2 · pp. 3–4

4.3 Verification Pipeline

  • Three independent judges must unanimously verify that target claims are inferable, no extra facts are introduced, and the document reads naturally.Human review corrected content in 11% of documents and removed scaffolding in 28%.Source: §4.3 · p. 4

5 Evaluation Framework

The framework evaluates memory with intrinsic metrics and downstream QA on the same trajectories, then measures how much their scores diverge. Intrinsic scoring compares the fact store with expected claims across trajectory states, while extrinsic scoring judges answers to questions about the final memory state.

5.1 Intrinsic Metrics

  • Intrinsic evaluation compares the fact store after a trajectory with expected claims at each state, treating entailment as embedding similarity above 0.8 followed by LLM-judge verification.Source: §5.1 · p. 4, §5 · p. 4
  • Intrinsic metrics include claim coverage, trajectory-level fact accuracy, update-detection recall for states requiring update or supersede, fact count, and confidence MAE.Fact count is used to detect duplication or loss.Source: §5.1 · p. 4

5.2 Extrinsic Metrics

  • Extrinsic evaluation asks diagnostic questions about the final memory state, retrieves relevant facts, generates answers, and has a judge score them against expected answers.Answers receive YES (1.0), PARTIAL (0.5), or NO (0.0).Source: §5.2 · p. 4

5.3 Divergence Metric

  • Mean absolute divergence (MAD) measures the gap between intrinsic and extrinsic scores, while divergence rate counts trajectories whose scores differ by more than τ = 0.15.MAD avoids an arbitrary threshold; divergence rate is the secondary interpretable measure.Source: §5.3 · pp. 4–5

6 Experiments

Across a representative memory system, intrinsic and extrinsic evaluation diverge, while direct inspection exposes operation-classification and fact-evolution failures that QA scores obscure. The experiments also show that divergence persists across system configurations and that diagnostic patterns vary with category and difficulty.

6.1 System Under Test

  • The tested representative system follows an extract-embed-classify-synthesize pipeline; intrinsic evaluation targets evolving synthesized facts rather than immutable artifact-level observations.The system retrieves related facts, classifies the relationship, and synthesizes the result into evolving facts with history.Source: §6.1 · p. 5

6.2 Results

  • 16.8 ± 2.5 percentage points is the mean absolute intrinsic–extrinsic divergence; 44% of trajectories exceed the 0.15 threshold.Under a ≥0.5 pass/fail criterion, the methods disagree in 21±4% of trajectories (κ = 0.118).Source: §6.2 · p. 5
  • At L1, intrinsic accuracy is 68.0 ± 3.2% versus extrinsic accuracy of 50.9 ± 4.2%; at L3, scores converge near 47% but diverge in opposite directions on individual trajectories.Classification accuracy declines across difficulty levels from 33.8% to 16.5% to 10.1%.Source: Table 3 · p. 6, §6.2 · p. 6
  • C5 temporal and C6 multi-author trajectories have the lowest intrinsic scores (59.3% and 41.5%), while extrinsic scores remain near-uniform across categories.C1 has the highest mean absolute divergence, 0.295 ± 0.084.Source: §6.2 · p. 6
  • Action-classification accuracy is 20.1 ± 2.7%, alongside a 22.2 ± 1.2% false-update rate and 73.6±2.6% update-detection recall.Classification accuracy is 4.5% for C2 and 0.4% for C5; the authors note the system can assign the wrong action even when its fact state is partly correct.Source: §6.2 · p. 6
  • Across four configurations, signed divergence stays positive, intrinsic accuracy remains in a 57.6–61.4% band, and extrinsic accuracy remains in a 48.5–49.2% band.System upgrades and the public-corpus switch move DivRate in opposite directions; divergence persists across configurations.Source: §6.2 · p. 7

6.3 Diagnostic Decomposition

  • From L1 to L3, claim miss rate rises from 58% to 88%, fact count from 9.1 to 20.7, and classification accuracy falls from 37% to 10%.The authors link poor classification to creating new facts instead of updating, fragmenting claims; extrinsic QA remains near 50% across difficulty.Source: §6.3 · p. 7

7 Discussion and Limitations

Intrinsic and extrinsic evaluation are complementary: one diagnoses memory state, while the other measures end-to-end utility. FEB-30 is limited to factual evolution across multi-author artifacts, with a modest trajectory count and judge-based scoring.

7 Discussion and Limitations

  • High scores on both evaluations indicate effectiveness, while a mismatch can signal LLM compensation or a retrieval or generation bottleneck.High extrinsic but low intrinsic scores signal fragility; high intrinsic but low extrinsic scores point to a retrieval or generation bottleneck.Source: §7 · p. 7
  • Generated documents may not capture every property of real multi-author text; the authors propose validating transfer against human-annotated document sequences.They mitigate this concern through style grounding, multi-model rotation, and hybrid injection at L3.Source: §7 · p. 7
  • FEB-30 does not evaluate episodic or procedural memory, or single-turn conversational recall, which existing benchmarks address.Its 30 trajectories support diagnostic evaluation but not fine-grained statistical comparisons; embedding thresholds and LLM judges may add measurement noise.Source: §7 · p. 7

8 Conclusion

FEB-30 enables direct inspection of expected memory states across fact-evolution trajectories. The evaluated system shows substantial divergence between intrinsic and extrinsic scores, and its claim coverage declines as trajectories progress.

8 Conclusion

  • 16 percentage points of average intrinsic–extrinsic divergence, 44% of trajectories above the meaningful threshold, and 20% correct memory-operation classification summarize the benchmark’s key findings.The conclusion reports operation classification falling from 34% at L1 to 10% at L3; Table 4 reports ClassAcc of 33.8±6.8 at L1 and 10.1±4.3 at L3.Source: §8 · pp. 7–8, Table 4 · p. 8

A.1 c5-linear: Linear Temporal Evolution

  • Intrinsic scoring checks whether each state retains earlier claims and records correct transitions, detecting fragmentation or dropped claims when they occur.In the c5-linear example, intrinsic scores are 40–67% while extrinsic scoring is 50%, and the latter masks partial fragmentation.Source: §A.1 · p. 9

A.2 c4-out-of-order: Out-of-Order Ingestion

  • The c4-out-of-order trajectory tests whether an older document enriches a later decision without overwriting it.The temporal guard should preserve the decision while adding why alternatives were rejected.Source: §A.2 · p. 10

B.1 Observation Extraction

  • The observation-extraction prompt is grounded in business/customer-relationship categories and structures each observation into who, what, when, why, and where fields.The statement refers to the “what” field.Source: §B.1 · p. 10

B.2 Fact Synthesis

  • Fact synthesis classifies observations as create, update, reinforce, supersede, or none, with updates required to preserve existing details and add new information.The prompt prefers updating over creating when an observation matches an existing fact.Source: §B.2 · p. 11

C.1 Claim Entailment Judge

  • The intrinsic judge checks whether active facts entail an expected claim, while extrinsic evaluation generates an answer from retrieved facts and judges it against an expected answer.The answer judge permits YES, PARTIAL, or NO assessments.Source: §C.1 · p. 11, §C.2 · p. 11, §C.3 · p. 11

D Models Used

  • The system under test uses one frontier LLM for observation extraction and fact synthesis, while separate models handle evaluation judging and answer generation.Retrieval combines BM25 keyword matching at weight 0.3 with k-NN vector similarity at weight 0.7.Source: D Models Used · p. 11

E Per-State Coverage Degradation

  • Claim coverage drops from 65% at s0 to 50% at s4, and longer trajectories correlate with lower intrinsic accuracy (Spearman ρ = −0.45, p < 0.02).Overall fact coverage is 98.8%, but created facts can be incomplete or fragmented; L3 starts at 50% claim coverage at s0.Source: E Per-State Coverage Degradation · p. 12

F Update Detection, False Updates, and Confidence Calibration

The system’s update detection and confidence calibration vary across diagnostic categories and difficulty levels.

F Update Detection, False Updates, and Confidence Calibration

  • C1 and C6 have both low update detection recall (∼51%) and high false update rates (∼39%), while C2 and C5 have near-perfect recall and zero false updates.This pattern indicates category-specific differences in classification performance.Source: F Update Detection, False Updates, and… · p. 12
  • Confidence calibration has an overall MAE of 0.106, ranges from 0.092 to 0.124 across categories, and is worst for C6 at 0.124.The reported C6 result is attributed in the passage to inadequate adjustment for author diversity.Source: F Update Detection, False Updates, and… · p. 12
  • Confidence MAE rises from 0.089 at L1 to 0.118 at L3, consistent with lower certainty on harder trajectories.The table caption notes that false update rate decreases with difficulty because the system rarely modifies facts at L3.Source: F Update Detection, False Updates, and… · p. 12, Table 9 · p. 12

G Metric Correlations

Across 30 trajectories, miss rate is strongly associated with intrinsic accuracy but only weakly with extrinsic accuracy; intrinsic and extrinsic scores share some signal but measure substantially different properties.

G Metric Correlations

  • Across 30 trajectories, miss rate correlates with intrinsic accuracy at r = −0.84 and extrinsic accuracy at r = −0.39, while intrinsic–extrinsic correlation is r = 0.55.The correlation pattern indicates that intrinsic and extrinsic evaluation are related but capture substantially different properties.Source: Table 10 · p. 12, G Metric Correlations · p. 12
  • Miss rate’s strong negative correlation with intrinsic accuracy supports claim coverage as the primary driver of intrinsic scores.The weaker association with extrinsic accuracy is consistent with LLM compensation masking extraction failures at query time.Source: G Metric Correlations · p. 12
  • C1 shows positive signed divergence, with intrinsic scores higher than extrinsic, while C6 shows no directional bias and both metrics are equally poor at multi-author attribution.Positive values mean memory is underestimated by QA.Source: G Metric Correlations · pp. 12–13, Table 11 · p. 13

H Robustness Across Systems: Configuration Comparison

Intrinsic–extrinsic divergence persists across four system and corpus configurations, and category-level rankings and named extremes remain stable under an upgraded backbone.

H Robustness Across Systems: Configuration Comparison

  • Positive signed divergence and near-chance agreement persist across all four configurations, so the finding is not limited to one system, corpus version, or backbone.The configurations vary system, corpus, or backbone; Main (released) is used for the paper’s other results.Source: H Robustness Across Systems:… · p. 13
  • Under the upgraded backbone, category rankings remain highly stable for MAD (ρ = 0.94), intrinsic accuracy (ρ = 0.89), and classification accuracy (ρ = 0.94).C1 retains the highest MAD and classification accuracy, C6 the lowest intrinsic accuracy, and C5 the lowest classification accuracy.Source: H Robustness Across Systems:… · p. 13
  • The upgraded-backbone comparison changes only adjacent rankings among already-close category values, without reordering the named extremes.The authors report this preserves the diagnostic category structure across backbone models.Source: H Robustness Across Systems:… · p. 13

I Per-Trajectory Results

Per-trajectory scores are reproducible across 11 runs, with intrinsic scoring more stable than extrinsic scoring and evaluating the persistent store directly.

I Per-Trajectory Results

  • Intrinsic scoring is more stable because it evaluates the persistent store directly rather than depending on retrieval and generation.Per-trajectory results are averaged across N=11 runs.Source: I Per-Trajectory Results · p. 13
  • Across 11 runs, intrinsic accuracy ranges from 56.9% to 63.3% (std = 1.6%), while extrinsic ranges from 45.0% to 55.0% (std = 2.9%).These ranges are reported as evidence of high reproducibility despite LLM non-determinism.Source: I Per-Trajectory Results · p. 13, Table 13 · p. 14
  • Table 13 reports trajectory-level MAD, miss rate, fact count, classification accuracy, and divergence direction; its direction symbols indicate which score is higher or whether the gap is under 5 points.Intrinsic above extrinsic denotes memory underestimated by QA, while extrinsic above intrinsic denotes LLM compensation.Source: Table 13 · p. 14
Loading 2610.22209v1…