When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model

1 Introduction

• The study asks whether one reranking call over raw turns is non-inferior to a strong extraction memory at a matched tight budget, defining non-inferiority as at most 5 points worse.The selector is Jev, a typed decision model; the extraction comparator is engram v2, and the comparison holds the selector fixed while changing the stored representation.Source: §1 · pp. 1–2
• Reranking’s gain fell from +17.4 to +1.5 points on LoCoMo and from +9.1 to +1.1 on LongMemEval as the answer model read 3 rather than 20 of 30 candidates.The authors offer this budget pattern as an interpretation of disagreement across published studies.Source: §1 · p. 2
• At the tight LoCoMo budget, the pre-registered test bounds extraction’s advantage to at most 4.7 points under the worst grading, with human-grading differences of −1.7 to −2.6 points.The test used held-out conversations, a second answer model, and blind human grading.Source: §1 · p. 2
• At matched context, Jev was non-inferior to a gpt-4o-mini listwise reranker (lower bound −2.0) at about a third of its latency, and more accurate than Jev-Mem’s graph traversal.The comparison is at answer level.Source: §1 · p. 2
• Diagnostics attribute 11.5% of questions to shortlist misses and 9.2% to rerank drops, with temporal evidence most often dropped at reranking; judge–human agreement also varies by system with answer length.The authors identify the answer-length interaction, rather than judge leniency itself, as their added finding.Source: §1 · p. 2
• Raw turns plus reranking is an established pattern (Derehag et al., 2026; Wu et al., 2026); the stated contribution is the test, within-study budget result, and typed selector.engram v2 is a version of the authors’ earlier system.Source: §1 · p. 2

Raw history against extraction

• Verbatim chunks outperform LLM-extracted artifacts by 15.9 points on LoCoMo and 22.0 points on LongMemEval-S in Fidelity Before Structure (An, 2026), while SmartSearch (Derehag et al., 2026) retrieves from raw history with learned ranking.Other raw-history approaches include retrieval and generation alone, near-verbatim discourse units, and chunk-based or mixed stores.Source: §2 · p. 2

Budgets and compression

• Kang et al. (2026) find consolidation helps when the token budget cannot fit relevant raw evidence, but retention is preferable once it fits; their intervention follows evidence identification, unlike this paper’s end-to-end selection test.They report gains of up to 48 points on LongMemEval at 32 tokens.Source: §2 · p. 3

Extraction systems

• Extraction systems use LLM calls to structure memory, while SimpleMem (Liu et al., 2026), LightMem (Fang et al., 2026), and LeanMem (Liao et al., 2026) target lower-cost compression, filtering, grouping, or content-specific records.The passage also describes mem0, Graphiti/Zep, A-MEM, MemGPT/Letta, EverMemOS, and Memora as structuring memory with write-time LLM calls.Source: §2 · p. 3

Typed decisions and reranking

• Prior typed-decision work includes Jev-Mem’s multi-graph operations (Jiang et al., 2026a) and reported Jev reranking gains; this paper compares a Jev reranker with an LLM reranker and Jev-Mem at matched context.Related reranking results vary: SmartSearch identifies ranking as a bottleneck, Fidelity Before Structure finds little reranking gain, and MemReranker (Li et al., 2026a) matches gpt-4o-mini on key retrieval metrics.Source: §2 · p. 3

Evaluation validity

• Evaluation choices shape retrieval claims: held-out LoCoMo splits predate this study’s pre-registration and non-inferiority margin, and retrieval credit can depend on stored form (Panthi and Abdelfattah, 2026).Fidelity Before Structure reports κ = 0.897 judge–human agreement; this study found agreement depended on answer style with its LoCoMo judge.Source: §2 · p. 3

3.1 Setup and notation

• A memory system writes a store from conversation turns and reads a question-specific subset; raw-turn systems store dated turns, engram v2 stores extracted facts with source quotes and validity windows, and Jev-Mem stores a graph.Each conversation is a sequence of turns with session dates.Source: §3.1 · pp. 3–4
• Jev answers a fixed-option typed question with probabilities in one request, choosing the most probable option and using that probability as confidence.Its closed-option scoring produces no generated text to parse, and the confidence can be thresholded.Source: §3.1 · p. 4
• Turns + Jev starts with a cosine shortlist of 30, ranks turns above relevance threshold 0.5, then appends a cosine-ranked floor of 10 turns before the answer model reads the first k.Turns + cosine instead reads the first k in cosine order.Source: §3.1 · p. 4
• H1 compares Turns + Jev with engram v2 question by question across 778 questions, declaring non-inferiority when the one-sided 95% lower bound clears the 5-point margin.Total per-question cost includes write cost amortized over 4.0 turns written per benchmark question, plus read and answer costs.Source: §3.1 · p. 4

3.2 The systems

• Turns + Jev stores dated raw turns without extraction or a write-time LLM call, then reranks shortlisted turns; Turns + cosine shares that store but uses cosine order, while Turns + LLM substitutes a gpt-4o-mini reranker.Full context places every conversation turn in the answer prompt; Figure 1 depicts the systems’ write and read paths and their calls.Source: §3.2 · pp. 4–5, Figure 1 · p. 5
• engram v2 extracts facts with an LLM and uses Jev to type, relate, and update them, while Jev-Mem uses two Jev requests per turn and two to sixteen per read.mem0 2.1.0 is also described as making one LLM extraction call per message and using vector search for reads.Source: §3.2 · p. 5
• A rule-selected temporal example shows engram v2 extracting the evidence turn under the wrong speaker and ranking three other facts above it at k=3, while Turns + Jev retained the dated verbatim turn.The example was selected by a fixed rule; an appendix example presents the opposite case, where the evidence turn missed Turns + Jev’s shortlist but appeared in engram v2.Source: §3.2 · pp. 5–6

• The plan was deposited before runs; an amendment after Batch A added S7, the full LongMemEval run, and a second answer model before any H1 result was seen.Source: §4 · p. 6
• Five LoCoMo conversations never run by any system supplied 3,122 turns and 778 scored questions, plus 209 adversarial questions; LongMemEval used 70 registered questions with user turns only.Source: §4 · p. 6
• At matched context, Turns + Jev used a k matched to the comparator’s natural k=3, selected using a retrieval-only token sweep saved before answers.Source: §4 · p. 6
• H1 tested judge labels against a margin half the development rerank effect; seven secondary tests used Holm correction and were mostly exact two-sided McNemar tests.Source: §4 · p. 6
• H1 and S1 were repeated with Llama 3.3 70B, while blind author grading audited H1 answers the judge rated discordantly.Source: §4 · p. 6
• All post-registration changes were dated; apart from the amendment, none changed a test, margin, or planned interpretation, though the mem0 serving deviation caveated S3.Source: §4 · p. 6

5.1 Primary test (H1, registered)

• The registered H1 test passed: Turns + Jev was non-inferior to engram v2, with a −0.5-point difference and a one-sided 95% lower bound of −3.0 against the −5-point margin.The comparison used 265 tokens per question for Turns + Jev and 251 for engram v2; human grading retained non-inferiority within the margin under every grading applied.Source: §5.1 · pp. 6–7, Table 1 · p. 7, Figure 3 · p. 8

5.2 Secondary tests (S1–S7, registered)

• At matched context, Turns + Jev beat similarity search, Jev-Mem, and mem0, was non-inferior to the LLM reranker (lower bound −2.0 points), and beat cosine by +9.1 points on full LongMemEval.The registered tests rejected S1, S2, S3, S4, and S7 after Holm correction; S5 and S6 detected no difference, which does not establish equivalence for the 30-question mem0 comparison.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 The budget dependence of reranking

• Reranking’s gain over similarity search shrank as k grew: +17.4 to +1.5 points on LoCoMo and +9.1 to +1.1 on LongMemEval from k=3 to k=20.The k=3 results were registered tests; k=20 differences were descriptive, and Turns + Jev’s score-scored read path often could not fill twenty slots.Source: §5.3 · p. 8, Figure 5 · p. 9

5.4 Robustness: a second answer model and blind human grading

• With Llama 3.3 70B answering from the same contexts, H1 still passed: Turns + Jev scored 75.3% and engram v2 75.6%, with a one-sided bound of −2.9.Blind human grading of judge-discordant answers moved the difference to −1.7 to −2.6 points and the bound to −3.9 to −4.7; under lenient grading, engram v2 was more accurate but remained inside the margin.Source: §5.1 · p. 7, §5.4 · p. 9

5.5 Long histories (LongMemEval)

• On LongMemEval, Turns + Jev scored 66.8% at k=3 versus 63.0% for full context, which read 209× the tokens at 29× the cost per question.Engram v2 was not run on LongMemEval, so these results do not show that Turns + Jev matches LLM-extracted memory on long histories.Source: §5.5 · p. 9

5.6 Where Turns + Jev’s accuracy stops

• On the five held-out conversations, evidence was absent from the 30-turn shortlist for 11.5% of questions and present but dropped by the rerank for another 9.2%.Open-domain evidence reached the shortlist least often (63.0%) and was dropped most often (31.4%); temporal evidence was dropped at reranking for 22.7% of questions, while multi-hop evidence was often incomplete in the shortlist.Source: §5.6 · p. 11

5.7 Cost and latency

• At the benchmark’s ratio of four turns written per question, Turns + Jev cost $0.00030 per question versus $0.00778 for engram v2.Jev read about 3.0× faster than the LLM reranker; extraction made engram v2 and mem0 thousands of times more expensive to write than Turns + Jev.Source: §5.7 · pp. 11–12, Figure 8 · p. 12

5.8 Abstention

• At k=3, correct abstention on adversarial questions fell from 63.6% with similarity search to 54.1% with Jev and 48.8% with an LLM reranker.The passages report that relevant-looking context makes the answer model less willing to say something was not mentioned.Source: §5.8 · p. 12, Table 6 · p. 12

6 Discussion

• Reranking’s gains are large when few candidates survive truncation and small when many are retained, supporting a budget-based interpretation of differing prior findings.This cross-pipeline interpretation is not itself tested across papers; the within-study k=3 to k=20 pattern supports it on two benchmarks.Source: §6 · pp. 12–13
• At k=20, extraction systems are more accurate and more compact than raw-turn selection in the measured comparison.engram v2 at k=20 was the most accurate system measured, with fewer tokens than Jev-Mem at k=40.Source: §6 · p. 13
• At matched context, Jev selects as accurately as the gpt-4o-mini reranker at about a third of its latency, and outperforms Jev-Mem’s multi-call graph traversal.Jev makes one request per question; the discussion attributes speed and cost, not higher accuracy, to the typed decision model.Source: §6 · p. 13
• Judge leniency interacts with answer length, producing different judge–human agreement by system; the discussion recommends reporting that agreement when answer styles differ.The discussion contrasts a generous grading instruction with a strict one as a possible explanation for differing audit results.Source: §6 · p. 13
• The paper’s best result is below SmartSearch’s reported 91.9%, but differing protocols make the figures non-comparable.The paper evaluates five held-out conversations under a different protocol.Source: §6 · p. 13

Limitations

• Five primary conversations provide 778 scored questions, with small category counts, limiting the breadth of the benchmark sample.The conversations are from LoCoMo, whose dialogues are LLM-generated.Source: Limitations · p. 13
• engram v2 was not run on LongMemEval, so the study cannot support claims about extraction on long histories.The study reports only narrower LongMemEval comparisons involving Turns + Jev, mem0, and Turns + cosine.Source: Limitations · p. 13
• The human audit has several constraints: the first author graded, partial-grade mapping was not prespecified, one question was ungraded, and only judge-discordant cases were re-graded.Judge errors on questions where systems agreed therefore remain unexamined.Source: Limitations · p. 13
• Adversarial content was not tested, although user-written text could shift Jev’s relevance decisions when raw turns are passed to it.The paper also notes Jev is a closed, versioned model and results apply to jev-1.13.0.Source: Limitations · p. 13
• mem0’s extraction calls were served through OpenRouter, about half by Azure, rather than the registered OpenAI API.Only S3 involves mem0.Source: Limitations · p. 13
• Turns + Jev (wide) was designed after the registered results and tested once on the same questions; every design choice was made on one development conversation.The reported accuracy is also below SmartSearch’s figures under a different protocol, so those absolute results are not directly comparable.Source: Limitations · pp. 13–14
• The k=20 comparison mixes budget effects with Turns + Jev’s read-path ceiling: its threshold and cosine floor yield 496 tokens versus 826 for Turns + cosine.A wider read path may make the budget dependence less steep, but that possibility is post-hoc.Source: Limitations · pp. 13–14

7 Conclusion

• Within this study, reranking added 17.4 points over similarity search on LoCoMo and 9.1 on LongMemEval when three of 30 candidates were kept (k=3); at k=20, gains were 1.5 and 1.1, respectively.Source: §7 · p. 14
• At a tight budget, the pre-registered test bounds extraction’s added benefit at no more than 4.7 points, and the result holds with a second answer model and blind human grading.Raw turns cost 3,061× less to write.Source: §7 · p. 14
• At matched context, Jev is non-inferior to an LLM reranker with a −2.0 bound at about a third of the latency, and more accurate than a multi-call Jev graph traversal.The conclusion characterizes the typed decision model as an effective selector.Source: §7 · p. 14
• Shortlist misses account for 11.5% of questions and rerank drops for 9.2%, with temporal evidence most often lost during reranking.The conclusion also reports that judge leniency interacts with answer length and recommends reporting judge–human agreement by system when answer styles differ.Source: §7 · p. 14

AI Assistance

• Claude Code assisted with code, run orchestration, and drafting under the first author’s direction.The first author made every methodological decision, approved each stage of the registered plan, and conducted the human audit.Source: AI Assistance · p. 14

Artifacts

• Code, plans, per-question answers, and judge labels are released with three tagged versions.The release also identifies the paper tag and result files.Source: Artifacts · p. 14
• The release includes human-audit grades and their answer key.The grades and key are stored with the human-audit results.Source: Artifacts · p. 14
• Plan documents and amendments, the paper, and the earlier engram preprint are deposited on Zenodo.The passage lists a separate DOI for each deposit.Source: Artifacts · p. 14

A Per-conversation results

• 77.2% versus 60.7% at k=3, while at k=20 Turns + Jev scores 76.6% versus 76.2% for Turns + cosine.These are exploratory results on four conversations and 610 scored questions.Source: A Per-conversation results · p. 17

B Registered plan and deviations

• The registered plan fixed systems, data, token matching, tests, predictions, human checks, run order, and budget before any run.Later changes were recorded as dated deviations; Table 8 is identified as the deviations table.Source: B Registered plan and deviations · p. 17, Table 8 · p. 17
• Two retrieval-only sweeps were accidentally stopped and then rerun from the cache.The paper notes this as an implementation detail outside the deviations table.Source: B Registered plan and deviations · p. 18

C Human audit

• −4.7 points was the worst human-grade non-inferiority bound, and H1 remained non-inferior under all four treatments of the ungraded question.The audit showed 142 judge-discordant questions to a blinded grader; retaining the judge’s labels for the ungraded item was the conservative treatment.Source: Table 9 · p. 18, C Human audit · p. 18

D Shortlist recall

• 76.9% of questions had all evidence in the shortlist, while reranking dropped all shortlisted evidence for 10.4% of fresh and 10.3% of exploratory questions.At least one evidence turn was in the shortlist for a reported share, but the passage does not provide those values here.Source: D Shortlist recall · p. 18

E Turns + Jev (wide), post-hoc exploratory

• 81.5% accuracy at 2,000 tokens was reported for post-hoc Turns + Jev (wide), with $0.00089 read cost per query and 720 ms p50 latency.The exploratory system used a 150-turn cosine shortlist and Jev reranking, with k=47 matched to Jev-Mem at k=40.Source: E Turns + Jev (wide), post-hoc… · p. 19

F Prompts and the Jev question

• Jev’s reranking question treats a turn as relevant when it states or directly implies part of the answer, not when it merely shares a keyword.The prompt also labels off-topic turns as false.Source: F Prompts and the Jev question · p. 19

G The mem0 serving incident

• A mem0 2.1.0 OpenRouter key silently routed all 3,122 extraction calls, splitting service between OpenAI and Azure and producing a reported $2.42 bill.The authors state that the serving difference cannot explain the much larger S3 difference; later runs pinned mem0 to OpenAI.Source: G The mem0 serving incident · p. 19

H Cost accounting

• All systems’ model and Jev calls are priced at list rates, while billed amounts are reported separately and read costs are per query.Totals per question state their assumed reads per write.Source: H Cost accounting · p. 19

I Reproduction

• Tables and figures are rebuilt from committed result files without API calls, and worked examples replay frozen stores through a read-only cache.A cache miss stops the worked-example replay rather than making an API call.Source: I Reproduction · pp. 19–20

J A counter-example

• A fact extracted from a turn about a nearby breeder reached the answer model at k=3, although that source turn missed the 30-turn shortlist.The figure contrasts the contexts at matched recorded token counts.Source: J A counter-example · p. 20, Figure 9 · p. 21