Auditing Long-Term Memory Evaluation: Repeated Judging, Reader Variation, and Negative Controls

1 Introduction

• Stages before the final answer use deterministic code, leaving the reader replaceable while making the same packets and downstream artifacts available for inspection.The report’s stated contribution is the documented evaluation procedure and its failure cases, not a new retrieval primitive; all 500 questions were used for development.Source: §1 · p. 2
• Single-run, LLM-judged scores separated by only a handful of questions can make leaderboard rankings partly an artifact of measurement noise.The motivating task requires retrieving facts scattered across long, sometimes superseded conversation histories.Source: §1 · pp. 1–2
• 475–479 correct answers were observed across two passes, a range containing Chronos’s published count of 478 but not a confidence interval or controlled comparison.The report pairs the observed range with repeated judging, per-question outcomes, and negative controls.Source: §1 · p. 2
• Three labels changed when unchanged answers were re-judged, reader lanes varied widely on fixed packets, and a verifier damaged more correct answers than it repaired.Lane differences also reflect tools and additional context, so they do not isolate a pure model effect.Source: §1 · p. 2

1.1 Related work

• The work positions its contribution as orthogonal to memory-store management and reports no claim of cross-benchmark generality from its recovered LoCoMo diagnostic.It situates retrieval in the retrieve-and-rerank tradition and connects its measurement procedure to prior work on LLM judging and single-run reporting.Source: §1.1 · p. 3

2 Benchmark and evaluation protocol

• LongMemEval-S has 500 questions over histories of roughly 40–60 sessions, including 470 answerable questions and 30 abstention questions.Evidence can span sessions recorded weeks apart; the study uses the cleaned release, whose maintainers describe it as preventing interference on answer correctness.Source: §2 · p. 3
• The adapted GPT-4o protocol modifies the knowledge-update template, and historical runs’ 72 knowledge-update answers were judged with that modified prompt.Other templates match the upstream version exactly or differ only by a trailing space; the report labels adapted-rubric scores separately from exact-upstream measurements.Source: §2 · p. 3

Selected historical comparison points (author-reported; not a current or exhaustive

• The pinned Chronos count is 478/500, but comparisons can cross data versions because Chronos does not specify whether it used the cleaned or original benchmark file.The report also notes that several higher quoted percentages are category-unweighted or use a different judge, while agentmemory V4’s 481/500 uses the evidence-only oracle variant rather than LongMemEval-S.Source: §2 · p. 3, Selected historical comparison points… · p. 4
• Agent Zero Memory reports 95.60% on 500 questions without naming the benchmark variant or judge, preventing classification as a like-for-like LongMemEval-S result.Published reader tiers also vary, so the report presents its strongest configuration alongside its reader ladder.Source: Selected historical comparison points… · p. 4

3 System architecture

• 468/470 questions had all gold sessions somewhere in the candidate pool, with 10 released answerable-packet gold sessions arriving through semantic expansion alone.The pool unions dense retrieval, keyword search, and a bounded semantic-expansion lane that adds candidates without removing or reordering them.Source: §3 · p. 4
• The chain’s first four stages are deterministic code, while stage five is a replaceable language-model reader scored by the adapted GPT-4o judge.Cross-encoder reranking scores candidate sessions against the question using the maximum chunk score; the figure identifies the chain’s staged flow.Source: §3 · pp. 4–5, Figure 1 · p. 5
• 462/470 answerable packets were gold-complete, and the shipped extension ordering achieved 462 versus 454 for the next-best policy on the same questions.The compiler preserves the dense top-10 as a prefix, fills remaining slots in cross-encoder order under a 16-session/40k-proxy-token budget, and passed determinism and negative-control checks.Source: §3 · p. 5
• For 240 of 470 answerable questions, deterministic scaffolds derive chronology, recency, candidate counts, and related evidence cues from only the packet and question.The scaffold is designed to be gold-blind and question-ID-independent; its source is held.Source: §3 · p. 5
• In the doctors example, the baseline retrieved none of the three evidence sessions, while semantic expansion supplied them and the final reader answered correctly.The evidence sessions occupied packet slots 13, 15, and 16; the scaffold enumerated candidate phrases but left judgments such as whether a planned procedure counts to the reader.Source: §3 · p. 6

4 Measurement methodology

• A headline requires two full passes to agree or a stated spread when they straddle the bar; the reported ± is half the range between passes.The rule was set before the Opus pair, and flip-row counts report run-to-run noise.Source: §4 · p. 6
• The second judge agreed on 493/500 rows in each headline pass and scored the pair 472/472, while adapted GPT-4o also gave opposite verdicts on an identical answer across runs.Observed judge disagreement limits ranking claims at margins this small.Source: §4 · p. 6
• A verifier that repaired 3 previously wrong rows was rejected after it flipped 11 of 58 correct drafts to wrong on the control replay.The frozen acceptance rule required zero control flips, preventing the verifier from reaching the headline result.Source: §4 · pp. 6–7
• The self-consistency result was withdrawn because it wrote one derived verdict into both passes and selected flip rows using judge-conditioned outcomes.Generic rules were revised after inspecting particular failures, which the report identifies as development-set tuning.Source: §4 · p. 7

5.1 Repeated LongMemEval-S measurements

• 479/500 and 475/500 are the Opus passes; a re-judge returned 478, while paired Opus–Grok tests establish neither superiority nor equivalence.Source: §5.1 · pp. 7–8

5.2 Per-question-type breakdown

• Against Chronos, Opus pass 1 is 479 vs. 478 (+1 net), while pass 2 is 475 vs. 478 (−3 net); the knowledge-update prompt differs.Source: §5.2 · pp. 8–9

5.3 Build history

• The v3.4 rung reports 476/474, but its +5/+4 gain spans earlier scaffold files and concurrent reader, operator, and prompt changes.Source: §5.3 · p. 9

5.4 Reader robustness

• Fixed-substrate reader lanes span 93 to 479; among the four strongest models scores span 465–479, but configuration differences preclude capability or substrate attribution.Source: §5.4 · p. 10

5.5 Recovered LoCoMo diagnostic: metric sensitivity, not a transfer claim

• +2.02 pp for Grok pass 1 full-response token-F1 became −0.04 pp on the last Answer line (95% CI [−1.18,+1.19]); extracted-answer intervals include zero.Source: §5.5 · p. 10

5.6 Knowledge-update template sensitivity on fixed answers

• The fixed-answer knowledge-update comparison uses 144 saved answers from 72 questions, with three judgments per template condition and no regenerated answers.The historical knowledge-update counts are 69/72 for pass 1 and 70/72 for pass 2; each of six matched repetitions credits one additional answer under the upstream template.Source: §5.6 · p. 11
• Across 432 matched template comparisons, six discordant pairs are upstream-only correct, all concerning a2f3aa27; repeated judgments are not 432 independent questions.For pass 2, the fresh modified-template count is 69/72 versus the historical 70/72, a judge drift on one question rather than a template effect.Source: §5.6 · p. 11
• The mixed sensitivity totals are 480/500 for pass 1 and 475/500 for pass 2, not new full-set re-judgings or replacements for the historical scores.The comparison measures rubric sensitivity on fixed outputs and does not resolve reader-context confounding or development-set tuning.Source: §5.6 · pp. 11–12

5.7 A clean-route reader on the frozen arm-D prompts

• The different-family clean-route pass scores 474/500, 1.0 percentage point below the headline pass, with a paired 95% interval of [−3.0, +1.0].Its live request bodies were not retained, so its complete input context cannot be verified; this comparison does not measure operator-context effects.Source: §5.7 · p. 12

5.8 The same clean reader on the baseline sessions

• +4.0 percentage points: the full package beats baseline sessions in the paired comparison, with a 95% interval of [+2.2, +6.0].The second judge also favors the full package by +3.4 points, with interval [+1.2, +5.6].Source: §5.8 · p. 13
• The baseline comparison used one pass per arm, and the served model may have changed between passes despite the shared requested reader label and route.Arms B and C, which would separate packet size, evidence index, and scaffold instructions, remain unrun.Source: §5.8 · p. 13

5.9 Evidence coverage of the two arms’ inputs

• 939/948 listed sessions and 491/500 questions are covered by full packets, versus 903/948 and 466/500 for baseline prefixes.Nine questions remain incomplete in the full packets, and 298 packets reach the 16-session cap.Source: §5.9 · p. 14
• 18 of 23 gains and none of 3 losses occur among 34 questions where baseline packets lacked listed evidence; the post-hoc split does not identify a component effect.On 466 baseline-complete questions, the arms are nearly level: 443 against 445.Source: §5.9 · p. 14

6 Error analysis: the grok pair’s remaining failures

• Of 470 answerable questions, 25 fail in at least one grok pass, with 18 wrong in both; bucket counts are approximate, not an exact partition.Some stable-wrong rows remain unassigned, and flip rows can change buckets between passes.Source: §6 · p. 14

6.1 Reader-cognition residue

• In about four reader-cognition residue rows, the packet has the correct evidence but the reader selects an evidence-backed distractor on question-boundary semantics.Examples concern whether a plan updates a location or whether an offered property counts as “viewed.”Source: §6 · p. 14

6.2 The verifier our own control rejected

• The cite-or-revise verifier repaired 3/12 winnable wrong rows but flipped 11 initially correct drafts to wrong, so its control rejected it.A second draft-blind design recovered 0/9 residue rows; both prompting-side repair designs failed under their controls.Source: §6 · p. 14

6.3 Gold-answer defects

• One strict gold defect says 15 weeks for events 81 days apart, while seven other gold defects are boundary cases.The $720 workshop row was reclassified as a boundary case: the packet states a $500 fee, but the disputed item falls after the question date.Source: §6 · pp. 14–15

6.4 Retrieval ceiling and coverage

• One row’s gold never reaches the candidate pool, and three in-pool gold sessions are lost to packet-budget ordering; alias expansion added no gold sessions.The remaining failures are flips or documented cases, with a few stable-wrong rows unassigned.Source: §6 · p. 15

No held-out evaluation

• Every score-moving component and the gold-defect inspection used the same 500 LongMemEval-S development questions; no held-out test-set number is reported.The recovered LoCoMo diagnostic does not resolve this threat because it is post-hoc, metric-sensitive, and uses unmatched treatment arms.Source: §7 · p. 15

Statistical resolution at the margin

• External leaderboard margins remain unresolved: nearby Wilson intervals overlap substantially, and like-for-like paired tests cannot be computed for the S-condition comparators.The headline was selected from tested reader configurations, and repeating the two-pass rule does not remove selection effects.Source: §7 · p. 15

Observed judging disagreement

• Three same-answer labels changed on re-judging, but two judges and one repeat do not estimate universal variance, and no independent human adjudicator checked verdicts or dossier classifications.The knowledge-update comparison also does not re-judge the other 428 rows or correct every historical lane.Source: §7 · p. 16

Reader-context audit

• The headline reader’s full input context cannot be established: transcripts show substantial operator material and available tools, but complete outbound requests were not retained.No effect of that context on accuracy was measured; the separate clean-route pass also lacked retained live request bodies.Source: §7 · pp. 16–17

Limited transfer evidence

• The released LoCoMo diagnostic supports no generality claim, while the packet-coverage figures describe the same development questions used to shape packet ordering.The report calls for a prospectively frozen, exposure-audited evaluation.Source: §7 · pp. 15, 17

Comparator and release provenance

• The release pins Chronos and Mastra comparator sources with SHA-256 digests, but OMEGA is not pinned and the manifest is unsigned.Hash agreement establishes consistency of released bytes, not execution authenticity, gold-blind construction, or method correctness.Source: §8 · p. 17

Released artifacts

• A fresh clone can verify manifest rows and re-derive headline counts without an API key, while saved-answer re-scoring requires an OpenAI API key and incurs usage charges.Reader outputs, judge verdicts, controls, and harness materials are released; retrieval, reranking, packet-compiler, and scaffold-operator sources are held.Source: §8 · p. 18

Reproducible checks and boundaries

• A third party can rerun the reader on released packets and scaffolds and rerun the judge on saved answers, but the headline reader’s complete input context cannot be reconstructed.Transcript-derived analyses are author-reported, and there is no independent end-to-end replication of the held stages.Source: §8 · pp. 17–18

Supplementary reader archive

• The supplementary clean-route archive supports recounting runs and re-scoring saved answers, but its live request bodies and prompt-building scripts were not retained.Repeating the reader configuration depends on a subscription login, client version, and floating served-model label, so an identical reader snapshot cannot be assured.Source: §8 · pp. 18–19

Transfer

• A transfer test needs a disjoint question set, documented exposure history, frozen operators, adapter checks, a pinned reader, and complete requests before outcomes are inspected.LongMemEval-M reuses the S questions, and LoCoMo has already been used, so neither is a pristine holdout.Source: §9 · p. 19

Component attribution

• The frozen A–D build ladder can estimate package, index, and scaffold contrasts, but B/C, a second pass, and a clean-route headline-model run remain outstanding.All 500 questions are development questions, so the comparison is diagnostic rather than a transfer result.Source: §9 · p. 19

Judge-sensitive rankings

• A one-answer gap from the pinned Chronos count is smaller than the three labels changed by re-judging unchanged answers, leaving which reporting standards clarify such rankings open.Candidate standards include per-question verdicts, repeated judging, and multi-judge agreement.Source: §9 · p. 19

Residual reader errors

• The next verifier candidate would require readers to cite packet lines for counted items and use a pre-specified control design sized to detect degradation.The remaining reader errors are question-boundary judgments.Source: §9 · p. 19

Reader thresholds

• Readers score the same packets from 93 to 479; mapping where performance collapses and whether scaffolds shift that threshold remains an open question.Source: §9 · p. 19

• 479 and 475 correct answers were reproduced across two adapted-judge passes, while re-judging unchanged answers changed three labels and yielded 478.Paired tests found no evidence of a difference between the two strongest observed lanes, but did not establish equivalence.Source: §10 · pp. 19–20
• +4.0 percentage points [95% interval +2.2, +6.0] separates the full package from baseline sessions under the same requested reader label, route, and judge snapshot.The post-hoc gain concentration on questions whose baseline packets lacked listed evidence does not separate packet, index, and scaffold effects.Source: §10 · p. 20
• A different-family reader scored 474, 1.0 percentage point below the headline pass, with a paired interval that includes zero.The prompts were rebuilt from the released packets and scaffolds; live headline requests were not retained.Source: §10 · p. 20
• Recovered LoCoMo results show that token-F1 gains do not survive answer-line extraction, underscoring the need to fix the scored answer representation in advance.Source: §10 · p. 20
• The remaining A/D comparison has one pass per arm, while B/C and repeats remain unrun; all questions were used in development and no untouched holdout was evaluated.The headline model on a clean route, an exposure-audited transfer set, and access to the held method also remain outstanding.Source: §10 · p. 20

• Grok scores 475/474 and Opus 474/471 on the 492 ex-dossier rows, but the comparison is conditioned on Grok’s errors and does not rank readers.The dossier was built from the Grok pair’s failures; Opus’s other failures were not audited for gold defects.Source: A Dossier sensitivity scenarios · p. 20
• Adjudicating the single strict defect raises the Opus pair to 480/476 and the Grok pair to 477/475, but a human adjudicator has not checked the classification.The two documented rows include one boundary row and one answer lost to fidelity, yielding net zero per pass before adjudication.Source: A Dossier sensitivity scenarios · p. 20

Inputs

• Arm D used one user message per question containing the question, compiled packet, and v2 instructions; 257 prompts also included an evidence index and scaffold instructions.The other 243 requests were identical to arm B’s, and the prompts included every session in the released headline packet.Source: B Clean-route reader pass: inputs,… · p. 20

Route

• The clean route used gpt-6.1-sol in an isolated configuration with no operator files, memory, plugins, hooks, MCP servers, or offered model tools.A required 66-character developer instruction preceded each frozen user message; the model slug was not a pinned snapshot.Source: B Clean-route reader pass: inputs,… · p. 21

Checks

• The live request body was not captured, so the offline twin—not the live-call record—supports checks on offered tools, verbosity, and input-item count.The live records confirm the frozen user text, model and effort labels, and absence of tool calls; a fixed hidden addition cannot be ruled out by the token count.Source: B Clean-route reader pass: inputs,… · p. 21
• The audit caught 12 of 12 planted defects across 12 of its 17 checks, but rules were narrowed and a stall detector corrected after false alarms.Audit alerts did not stop the live reader during this pass; only runner per-call checks could.Source: B Clean-route reader pass: inputs,… · p. 21

Judging

• The GPT-4o judge scored in two batches with positive and negative controls; no row was judged twice after the second batch resumed from an account-credit stop.A second judge scored answers as they landed, but it was not a like-for-like copy and no verdict was checked by a human.Source: B Clean-route reader pass: inputs,… · pp. 21–22
• The 72 answerable knowledge-update answers scored 70 under the exact upstream template, with controls passing 12/12 positive and 12/12 negative.Source: B Clean-route reader pass: inputs,… · p. 22

Arm A

• Arm A’s 494 new calls used an exact token count of 688 per call; two mechanically repeated questions received identical second answers, with the first answers kept unscored.The pass stopped and resumed twice, including once at a preset usage ceiling; no answer was dropped, edited, or regenerated.Source: B Clean-route reader pass: inputs,… · p. 22
• Arm A’s 71 new answerable knowledge-update answers scored 70 under both the released and upstream templates, while its model slug was not pinned.Its 494 new answers were judged in one batch; six reused arm-D answers retained their arm-D verdicts.Source: B Clean-route reader pass: inputs,… · p. 22