PaperlayerView exact revision on arXiv

Source-linked AI summary

Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate

Xupeng Chen, Chenqian Le

arXiv:2610.15710v1cs.AI

TL;DR

Memory-retirement benchmarks need to distinguish established change from a hint, since a gate can delete true facts when it treats suggestion as confirmation. This paper defines a three-way contract and evaluates Jev: its binary gate retires on hints, while the three-way choice reduces that error at too great a detection cost and behaves like a threshold shift.

  • Problem

    Existing memory benchmarks evaluate whether a pair is changed but do not separate established change from a hint, although confusing them can delete true stored facts.

    Source: §1 · p. 1

  • Method

    The authors define a three-way retirement contract, label STALE pairs and same-user controls, and evaluate Jev decision variants under pre-registered hypotheses.

    Source: §1 · p. 1, §3 · p. 3, §4 · p. 4

  • Results

    39.2% of NEEDS-CONFIRMATION claims are retired by the binary gate; the three-way Choice reduces this to 15.5% but loses 15.3 points of detection, exceeding the pre-registered budget.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Memory systems should treat a positive gate decision as evidence to attach, not permission to delete, unless the decision model has been scored on suggested changes.

    Source: §9 · p. 7

    The vault’s shared-error analysis is exploratory because all vault human labels come from the second annotator, and an annotator independent of both model families would settle it.

    Source: §8 · p. 7

Abstract

from arXiv · show

Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair as changed or not, so they cannot tell a gate that acts on established change from one that acts on a hint. We write a task contract that separates the two (SUPERSEDED vs. NEEDS-CONFIRMATION vs. COMPATIBLE), label 500 test pairs built from STALE (change pairs plus same-user controls) with three LLM annotators and two human adjudicators (one author and a second annotator; provenance reported), and evaluate Jev, a typed "System One" decision model, under pre-registered hypotheses. Of the 179 test pairs STALE marks as changed, 72 meet the contract's bar for an established change. Jev never retired a claim on 141 same-conversation non-changes. Its failure is premature retirement: the binary gate retires 39.2% of claims whose change is only suggested. A three-way decision cuts this to 15.5% at a 15.3-point detection cost, which misses our pre-registered 10-point budget, and it lands on the binary gate's own threshold frontier; these verdicts hold under three readings of the gold labels. On a held-out vault scored against a second annotator's labels, a cascade that escalated uncertain cases to a reasoning model and looked best on development data was indistinguishable from simply raising the threshold (premature retirement 26.8% vs. 6.5% on dev). Part of the gap comes from the gold labels: relative to that annotator's reading, LLM-majority gold shares most of its errors with a same-family escalation model, and an identity for binary labels says how much such shared errors hide. The rest is small-sample selection on development data. For builders this means a binary score should not be wired to delete: route suggested changes to pending evidence, and do not trust a cascade selected on development data of this size.

1. Introduction

Memory-retirement evaluation must distinguish established changes from hints, because deleting on a hint can erase a true stored fact. This paper defines that distinction and evaluates Jev on the retirement decision rather than a full memory system.

1. Introduction

  • The study evaluates one retirement decision under a written contract and pre-registered hypotheses, rather than building or comparing memory systems.This decision remains necessary in systems such as Jev-Mem, which keeps observations and defers contradiction and obsolescence to maintenance.Source: §1 · pp. 1–2
  • The binary Jev gate retires 39.2% of suggested changes, while its three-way decision reduces this to 15.5% but acts like a threshold shift.The three-way decision loses 15.3 points of detection, exceeding the pre-registered budget.Source: Figure 1 · p. 2, §1 · p. 2
  • The task contract separates established change (SUPERSEDED), suggested change (NEEDS-CONFIRMATION), and no change (COMPATIBLE), with a corresponding memory action for each.The relabelled STALE sample includes natural same-conversation controls.Source: §1 · p. 2

2. Task and estimands

The task contract asks what a later utterance establishes about a stored claim, separating confirmed replacement from a hint and compatibility. It also specifies the action expected for each label and the rates used to assess a gate.

2. Task and estimands

  • SUPERSEDED requires the same subject, attribute, and scope; a current or completed assertion; and a value incompatible with the stored claim, without unstated assumptions.If reaching SUPERSEDED requires writing “assuming...,” the label is NEEDS-CONFIRMATION.Source: §2 · p. 2
  • A memory system retires the claim only on SUPERSEDED, attaches the new utterance as pending evidence on NEEDS-CONFIRMATION, and does nothing on COMPATIBLE.Source: §2 · p. 2
  • The evaluation estimates detection on SUPERSEDED items, premature retirement on NEEDS-CONFIRMATION items, and false invalidation on COMPATIBLE items.Each rate is reported with a Wilson 95% interval.Source: §2 · p. 2
  • The pre-registered safety criterion requires F ≤.05 pooled and within-session and P ≤.20; the three-way-help hypothesis allows at most a 10-point detection loss.Binary gates retire at τ = .5, while three-way gates retire only on SUPERSEDED; strata with n < 15 are descriptive.Source: §2 · p. 3

3. Data and gold labels

The test and vault are scenario-split samples from STALE, augmented with same-user controls and labelled under the three-way contract. The test gold combines LLM majority labels with human adjudication, whose provenance and limits are reported.

3. Data and gold labels

  • The 500 test pairs comprise 179 STALE change pairs, 179 cross-session neighbours, and 142 within-session neighbours; the vault contains 176 pairs.The within-session neighbour was added before annotation and is the primary false-invalidation control because cross-session neighbours were mostly off-topic.Source: §3 · p. 3
  • Three blind LLM annotators labelled each pair, and test gold is their three-way majority except where human adjudication applies.Agreement on change pairs was weakest between A1 and A2 (κ 0.28) and A1 and A3 (κ 0.22); most disagreement concerned the SUPERSEDED/NEEDS-CONFIRMATION boundary.Source: §3 · p. 3
  • The second annotator supplied the gold labels for 68 items after a model fill; the authors report these as a reading confirmed by two people, not an independent human reading.The vault analysis depends on these labels and is therefore exploratory; the test verdicts do not.Source: §3 · p. 3
  • Of 179 test change pairs STALE calls changed, 72 are SUPERSEDED under the contract; only 13 of 90 Type II pairs meet that bar.The contract excludes Type II propagation by its same-attribute condition, so the predominance of suggestions among Type II pairs is partly definitional.Source: §3 · pp. 3–4

4. Systems

The systems include lexical and embedding baselines, binary and three-way Jev variants, and a reasoning-model comparator. Test outputs were sealed until scoring, and the Jev variants differ in decision format and label naming.

4. Systems

  • S0 uses a change cue and word-overlap cutoff, while S1 retires when MiniLM cosine similarity between the claim and new utterance is at least .5.Both cutoffs were fixed before outputs were read.Source: §4 · p. 4
  • S2 is Jev as a binary Noul using an existing archive-change judge’s wording; S3 is a three-way Choice developed on development data.S3n uses neutral option letters, while S3x assigns definitions another label’s name; decisions are read from the chosen definition.Source: §4 · p. 4
  • S4 is Claude Sonnet 5 with the contract text and is run on change pairs only; systems were frozen and test outputs remained sealed until first scoring.Gold was later revised twice for provenance reasons without system outputs, and no system was rerun or verdict changed.Source: §4 · p. 4

5. Pre-registered results on test

On test, binary Jev retires suggested changes too often, while the three-way Choice reduces premature retirement at a detection cost above the pre-registered budget. The three-way decision lies on the binary gate’s threshold frontier, and the reported H1 and H3 verdicts hold across three gold-label readings.

5. Pre-registered results on test

  • No Jev variant retired any of the 141 same-conversation COMPATIBLE controls, but binary Jev still prematurely retires 39.2% of suggested changes.Pooled false invalidation is low because most COMPATIBLE items are controls.Source: §5 · p. 4
  • 39.2% of NEEDS-CONFIRMATION claims are prematurely retired by binary Jev, versus 15.5% by the three-way Choice, which loses 15.3 points of detection.The detection loss exceeds the pre-registered 10-point budget, so H3 fails.Source: §5 · p. 4
  • The three-way Choice sits on the binary gate’s threshold frontier: S2 at τ=.65 reaches 79.2%/17.5%, versus S3 at 77.8%/15.5%.This threshold comparison was not pre-registered.Source: §5 · p. 4, Figure 2 · p. 5
  • Every H1 and H3 verdict remains unchanged under all three gold-label readings.With the LLM-majority reading, S3 prematurely retires 16.7% of NEEDS-CONFIRMATION items; with no human labels, it retires 17.0%.Source: §5 · p. 4
  • H2 is not testable: only 11 eligible SUPERSEDED items are implicit or numeric, and the contract sends most implicit changes to NEEDS-CONFIRMATION.The authors say an implicit-change test needs a different sample.Source: §5 · p. 4
  • Changing S3’s option names changes decisions without materially changing ranking: neutral and permuted names alter 3.6% and 10.2% of decisions, respectively.Detection falls from 77.8% to 51.4% with permuted names, while AUROC stays within .01 of S3’s.Source: §5 · p. 4
  • Label-free prior division does not remove the option-name effect: S3x still disagrees with S3 on 11.4% of items and detects 59.7% against S3’s 80.6%.The authors report that the effect varies by item rather than acting as a fixed per-option offset.Source: §5 · p. 4
  • The reasoning-model reference S4 is at parity with S3 on change pairs, at 80.6%/14.6%.S4 ran on change pairs only; its premature-retirement and false-invalidation rates use different denominators from the full-sample columns.Source: §5 · p. 4, Table 1 · p. 5

6. Held-out evaluation and why development overestimated

On the held-out vault, the development-selected cascade did not outperform raising the binary threshold. The authors attribute the development-to-vault gap partly to shared gold-label errors and partly to small-sample selection.

6. Held-out evaluation and why development overestimated

  • On the vault, M4 and M5 both reached 95.0% detection with 26.8% premature retirement, matching each other and exceeding M4’s 6.5% dev premature-retirement rate.They differed on only six items, split evenly; the authors say a smaller advantage might not be visible on this vault.Source: §6 · p. 5, Table 2 · p. 6
  • Against second-annotator labels, M4’s vault premature retirement was 11/41, versus 7/37 (18.9%) against LLM-majority gold; development selection used only 31 NEEDS-CONFIRMATION items among 26 candidates.M4’s dev rate was 2/31, but its one-sided 95% upper bound was 18.9%, above the .15 target; certifying .15 with two errors requires at least 40 such items.Source: §6 · pp. 5–6
  • Error against gold understates true error exactly when a system and gold share more than half of the gold’s errors.The identity applies to binary decisions, collapsing the three labels into retire or keep.Source: §6 · p. 6
  • On 27 vault band items, the escalation model and majority gold shared 7 errors relative to the second annotator’s labels, hiding a net 3 errors out of 10.The shared-error diagnosis is exploratory: the second annotator supplies only one reading, and the data cannot determine which model-family labels are wrong.Source: §6 · p. 6

7. Related work

Related work studies memory updating, fast typed decision models, and LLM-based evaluation. This paper distinguishes deleting an earlier claim on sufficient evidence from recovering the latest value in downstream answers.

Memory update and forgetting

  • Memory benchmarks assess whether assistants recover the latest state, while this paper scores whether an earlier claim was deleted on sufficient evidence.Mem0’s delete branch is a retirement gate; A-MEM links and evolves notes, and STALE isolates the update step.Source: §7 · p. 6

Fast typed decision models

  • Jev-Mem uses Jev as a memory control plane but has no retirement operation, while AnyJev adapts the interface to open-weight models and evaluates on JevBench’s released expected answers.The paper reports a smaller option-name effect on its new task, with probability ranking nearly unaffected; prior division fixes position bias but does not fix the name effect.Source: §7 · p. 6

Evaluation with LLM labels

  • LLM annotators and judges are common, but prior work warns of same-family preference; this paper contributes a pre-registered fallback and an exact decomposition of shared-error hiding.The fallback measured the issue as 8/20 overturns and required human labels where it mattered.Source: §7 · p. 7

8. Limitations

The evaluation has several scope and provenance limits: human labels are individual readings, vault comparisons are exploratory, and the tested system and benchmark are narrow.

8. Limitations

  • Each human-labelled item has one adjudicator, so those labels are one reading rather than a consensus.The vault labels all come from the second annotator, making the shared-error finding conditional on that reading and exploratory.Source: §8 · p. 7
  • The three-way Choice gate was not run on the vault, and its test premature-retirement rate of 15.5% remains unreplicated.The vault shift for S2 at τ=.7 suggests the result might not hold; an independent vault labeling is also still needed.Source: §8 · p. 7
  • Results cover one pinned Jev version on one English benchmark, and do not establish whether findings generalize to other models or update benchmarks.Cost, latency, and end-to-end effects inside a memory system were not analyzed.Source: §8 · p. 7

9. Conclusion

The conclusion is that a binary memory gate can retire claims on hints, while a three-way decision reduces that risk only through a threshold-like trade-off. The authors recommend separating established from suggested change and reporting how many system errors gold labels can see.

9. Conclusion

  • The binary gate retires 39.2% of suggested changes, while the three-way decision cuts this to 15.5% as a threshold shift rather than new information.The gate was safe on same-conversation non-changes, but unsafe on hints.Source: §9 · p. 7
  • Memory systems should treat a positive gate decision as evidence to attach, not permission to delete, unless the model has been scored on suggested changes.The conclusion also calls for benchmarks to label established and suggested change separately because STALE’s “changed” category includes both.Source: §9 · p. 7
  • Evaluations using LLM-majority gold should report, under Proposition 1, how many system errors the gold can see.The identity specifies when gold-based error understates true error: when more than half of gold’s errors are shared with the system.Source: §9 · p. 7

Impact Statement

The paper evaluates whether a fast decision model can safely retire stored user facts, while documenting its data, annotation, and proof procedures. Annotator detection varied by conflict type, and the paper states that the human labels are individual readings rather than consensus.

Impact Statement

  • The study uses public-benchmark text pairs; its only human participants were one author and one annotator, each labeling pairs from that benchmark.The authors report no affiliation with or funding from TypeSafe AI; Jev was accessed through its public API.Source: Impact Statement · p. 7

A. Reproducibility Statement and Use of AI Assistants

  • Scripts generate every reported number, table, and figure from saved annotator and system outputs; the authors say the inputs, outputs, dated contract, hypotheses, and code will be released upon publication.The two agent pre-fill exports are also slated for release because the repository cannot regenerate them.Source: A. Reproducibility Statement and Use… · p. 10
  • AI assistants supported experiment code, public-data collection, citation checks, and editing; an AI assistant drafted Proposition 1 and its proof, which authors and another assistant checked.LLMs also served as annotators.Source: A. Reproducibility Statement and Use… · p. 10

B. Proof of Proposition 1

  • The proof gives a binary-label identity showing that measured error is smaller than true error exactly when shared disagreements with H are sufficiently frequent.The identity applies after collapsing the three labels into retire or keep.Source: B. Proof of Proposition 1 · p. 10

C. Annotator agreement

  • Figure 3 compares annotators’ retire-or-keep agreement with the second annotator on 37 vault items, and S4 on the 27 items in M4’s band.The figure reports Wilson 95% intervals.Source: Figure 3 · p. 10
  • S2 detected 96.6% of 59 Type I SUPERSEDED items and 76.9% of 13 Type II items; S3 detected 81.4% and 61.5%, respectively.Type II is descriptive because n=13, below the stated threshold of 15.Source: C. Annotator agreement · p. 10

D. Method-development round and model identifiers

The development round compared prewritten model candidates and thresholds, then identified the cascade and its band before reasoning-model outputs were available. Model identifiers and run dates specify the systems and evaluation timeline.

D. Method-development round and model identifiers

  • Candidates included five-Noul M1, S2-plus-M1 M2, mean-probability M3, re-thresholded S2 M5, and cascade M4; M1 retired almost nothing on development data.Post-hoc M1b, M2b, and M6 variants did no better than M5; the cascade and its band were specified before reasoning-model outputs on development data.Source: D. Method-development round and model… · p. 11
  • S2, S3, S3n, and S3x were Jev 1.13.0; S4 and A1 were Claude Sonnet 5, while A2 used same-family Claude subagents and A3 used GPT-6 Astra.Development and test ran September 23–24, 2026; vault runs followed September 27, and the second annotator labeled September 28.Source: D. Method-development round and model… · p. 11

Related papers

Loading 2610.15710v1…