TWIST: A Proposed Benchmark for Intervention Quality in Conversational Memory, with a Human-Validated Draft-Alignment

1 Introduction

• Failures of revision can occur even when storage, retrieval, and answering work correctly, including stale beliefs, contradictory drafts, and inappropriate disclosure.Such failures can carry concrete costs, including lost renewals, misleading emails, stale-data decisions, and failed security reviews.Source: §1 · pp. 1–2
• TWIST measures intervention quality across four tracks: unprompted tension detection, draft alignment, belief supersession, and safe recall.The suite extends LoCoMo’s corpora and harness, so one run yields a recall score and a TWIST profile.Source: §1 · p. 2
• TWIST treats benchmark validation as part of the task, using independent gold-blind double annotation with adjudication, judge decoy calibration, and a separability audit.Track B pairs matched hard negatives with a gold-evidence detectability check.Source: §1 · p. 2
• The paper instantiates and human-validates Track B, while Tracks A, C, and D are specified for a later release.Reference results on the frozen v1.0 key and per-item outputs are released for a deployed coherence-oriented system and flat-RAG baselines across three backends.Source: §1 · p. 2
• Every detect/block metric is paired with surface-matched hard negatives, requiring systems to demonstrate restraint as well as intervention.The paired controls prevent a system from succeeding by flagging everything.Source: §1 · p. 2

Recall benchmarks

• LoCoMo and LongMemEval evaluate recall or prompted knowledge updates, but neither requires unprompted contradiction detection, outgoing-draft vetting, or write-time governance.LoCoMo-J uses an LLM-as-a-judge convention; an independent audit found 6.4% of its gold key score-corrupting and its judge accepted 62.8% of deliberately wrong answers.Source: §2 · pp. 2–3

Experience-reuse benchmarks

• Evo-Memory tests experience reuse across evolving task streams, whereas TWIST evaluates whether memory remains coherent, current, and governable as described events change.The paper frames recall, reuse, and revision as complementary benchmark dimensions.Source: §2 · p. 3

Belief dynamics and evolving state

• Evolving-state benchmarks cover belief tracking, memory safety, supersession QA, forgetting, temporal disambiguation, and relational discrimination, but at different system layers and task scopes.TWIST emphasizes intervention tasks with evidence attribution and matched false-intervention controls at the memory-system layer.Source: §2 · p. 3

Intervention quality as the evaluation target

• TWIST relates its controls to existing anti-update and safety controls while applying matched restraint tests to tension flags, draft vetting, and disclosure governance.The paper notes that its initial public release specified four tracks but instantiated only Track B; release chronology does not establish when projects were conceived.Source: §2 · p. 3

Memory systems with coherence machinery

• Existing systems include conflict detection, bi-temporal invalidation, belief-revision semantics, and draft vetting; TWIST adds an intervention-focused evaluation protocol.Its reporting rules require versioned keys and per-item outputs for reproducible comparisons.Source: §2 · pp. 3–4

3 Task tracks

• Track B evaluates draft alignment, Track C tests current-belief answers with retained history, and Track D measures safe recall; TWIST comprises four tracks over two corpora.Systems implement a five-call API and may decline a track, reported as N/A rather than zero.Source: §3 · p. 4

3.1 What counts as a tension: a belief-transition taxonomy

• TWIST scores unresolved belief transitions rather than logical contradiction alone, distinguishing contradiction, supersession, acknowledged revision, coexistence, and uncertain updates.Acknowledged revision is a hard negative for Track A but a supersession for Track C; coexistence requires retaining both facts.Source: §3.1 · p. 4

3.2 Track A — Unprompted tension detection

• Track A detects a speaker’s unprompted stance reversal across sessions and pairs each gold arc with a surface-matched non-contradiction.It measures detection F1, false-flag rate, latency in online mode, and attribution; duplicate flags and excess evidence do not increase scores.Source: §3.2 · p. 4

3.3 Track B — Output-time draft alignment

• Track B vets an outgoing draft against the record, reporting contradiction recall, aligned specificity, hard-negative specificity, and evidence attribution.Grounded contradiction recall requires a correct flag with valid top-three evidence; the track defines no aggregate score.Source: §3.3 · pp. 4–5

3.4 Track C — Belief supersession

• Track C tests whether systems answer with the latest fact, preserve superseded facts as ordered history, and respond to explicit resolution.Its metrics include current-answer rate, stale-answer rate, history retention, and resolution responsiveness; coexisting facts serve as hard negatives.Source: §3.4 · p. 5

3.5 Track D — Safe recall

• Track D tests sensitive recall, including PII, crisis content, and emotional hard negatives that should remain recallable.The v2 specification adds authorization-scoped recall paired across unauthorized and authorized access, and a separate storage audit for systems exposing an auditable state interface.Source: §3.5 · p. 5

3.6 System-under-test API

• The API covers ordered ingest, recall, draft-alignment checks, tension flags, and an optional resolution hook, without assuming a particular architecture.Track D v2 additionally specifies authorization-scoped recall and a storage-audit interface.Source: §3.6 · p. 5

4.1 Injection methodology (LoCoMo track)

• The LoCoMo track injects arcs, drafts, update chains, and safety plants into 10 existing conversations, preserving comparability and surrounding conversational noise.Injections add turns within existing sessions and up to three appended sessions; original QA keys are revalidated, and perturbed questions are versioned out.Source: §4.1 · p. 6

4.2 TWIST-CS: a multi-channel workspace track

• Six synthetic multi-channel workspaces contain 400–900 events each and model channel switching, multiple speakers, and business arcs absent from the LoCoMo personas.The CS track is designed to measure the added difficulty of evidence scattered across channels and interlocutors.Source: §4.2 · p. 6

4.3 Item-quality rules

• Five item-quality rules require self-contained evidence, verification from the system-visible excerpt, paired hard negatives, frozen held sets, and history-length stratification.These rules respond to violations observed in the v0 pilot.Source: §4.3 · p. 6

4.4 The Track B v1 generation pipeline

• Structural gates reject role-flip items, absence-based contradictions, and malformed evidence ids before an LLM verifier checks dated standalone excerpts.Contradicting items must include a verbatim contradicted quote; the verifier must judge both contradiction and claim-stated status from the evidence as seen.Source: §4.4 · p. 6
• Of about 490 generated items, 200 survived; audits found the gate’s rejections overwhelmingly correct and identified role-flips, absence-based contradictions, and question-form drafts as major defects.Undated excerpts also prompted verifier false rejections, motivating the date requirement; rejected items and verdicts are released in a sidecar file.Source: §4.4 · p. 6

4.5 Composition

• Track B’s final validated key contains 161 items: 38 contradicting, 61 aligned, and 62 hard-negative items.The original 200 candidates covered histories of 369–689 turns; hard negatives deliberately exceeded the proposed share because they distinguish systems.Source: §4.5 · p. 7

5.1 Answer-key integrity

• κ = 0.851 on the 161-item frozen TWIST-v1.0 key, above the preregistered 0.8 bar after adjudication and quality filtering.Candidate-generation agreement was κ = 0.565; this is the unbiased reliability estimate, while 0.851 is conditioned on adjudicated selection. The three-LLM pre-screen was triage, not a substitute for human annotation, and an independent third-annotator pass is planned for the venue version.Source: §5.1 · pp. 7–8

5.2 Judge robustness

• Tracks A–C use evidence-id overlap where possible; unavoidable LLM judges are tested against decoys and must reject at least 95%.Track C reports class-wise decoy rejection because leniency toward stale answers can inflate its current-answer metric; Track B uses structural matching rather than an LLM judge.Source: §5.2 · p. 8

5.3 Separability audit

• Given cited gold turns, all three Track B backends reach 1.000 contradiction recall, showing every kept item is detectable from its own evidence.This detectability oracle bounds neither specificity nor full-record reasoning; flat-RAG performance and human validation provide the Track B audit context.Source: §5.3 · p. 8

5.4 Reproducibility standards

• The paper publishes the frozen key, annotation and adjudication records, council pre-screen, per-item outputs for all thirteen configurations, and statistics.Scores must cite the key version, which is maintained through a public errata process.Source: §5.1 · p. 7, §5.4 · p. 8

• TWIST reports a five-column profile—LoCoMo-J, A detection F1/false-flag, B contradiction recall/hard-negative specificity, C current/stale, and D leak/over-block—with no composite score.Paired metrics must be reported together, and online and batch modes occupy separate leaderboard columns.Source: §6 · p. 8

7.1 Pre-registered predictions

• The pre-registered Track B prediction that flat RAG would catch blatant contradictions but fail hard negatives was confirmed across backends, with contradiction recall of 0.76–0.97.Separation was confirmed for specificity and attribution but refuted for raw detection recall, where flat RAG dominates; the authors note model and architecture are confounded in most comparisons.Source: §7.1 · p. 9

7.2 Pilot: Track B v0

• Across the 153-item pilot, flat RAG flagged every true contradiction but also 63% of hard negatives, while the deployed system caught only 11–14% and never over-flagged.The pilot exposed a role-flip defect: in most residual misses, the gold evidence turn alone did not carry the contradicted claim.Source: §7.2 · p. 9

7.3 v1.0 results

• Gold-evidence recall is 1.000 for all three models, while full-context results for calibrated models are close to solving Track B, consistent with substantial retrieval-coverage limitations.For Gemini, flat-RAG recall is 0.76 versus 0.92 with full context; the authors caution that full context also changes chronology and disambiguation cues.Source: §7.3 · p. 10
• Context effects on hard-negative specificity vary by model: Claude reaches 0.92 with the full transcript versus 0.50 with gold excerpts, while Gemini stays at 0.84–0.90.GPT-4o is over-sensitive across conditions, with specificity of 0.37 under full context; draft-only floors also vary, from 0.03 for Gemini to 71% contradiction flags for GPT-4o.Source: §7.3 · p. 10
• Across re-ingestions, deployed-system recall averages 0.39 with a 95% interval of [0.19, 0.51], and GPT-4o flat-RAG hard-negative specificity averages 0.48 [0.37, 0.66].Attribution and grounded-recall variance are estimable only from the cap-consistent ingest; the authors identify a three-fresh-ingest protocol as a venue upgrade.Source: §7.3 · p. 10
• With longer histories, deployed-system recall falls from 0.48 to 0.31, while full-context Claude recall is essentially length-robust at 0.92→1.00.The strata are small, including n=13 long-history contradicting items, so the authors report direction rather than significance.Source: §7.3 · p. 10
• On the frozen TWIST-v1.0 key (n=161), no tested configuration combines high contradiction recall, high hard-negative specificity, and high grounded recall; deployed recall is 0.421, with specificity 1.000 / 0.984.Source: §7.3 · p. 10, Table 3 · p. 11

8 Limitations and ethics

• Real-workspace validation remains future work with consenting design partners because items are LLM-drafted and naturalness depends on generation quality.Mitigations include persona-matched generation, structural gates, human annotation, and structural realism for the CS track.Source: §8 · p. 11
• Track B v1 items share one generator family, GPT-4o, so systems from that family may share blind spots with the item distribution.Model-free structural gates, a two-family council pre-screen, and human annotation are disclosed mitigations.Source: §8 · p. 11
• Track D crisis plants use non-graphic test phrasings in an opt-in, warned file; the corpus measures governance, not crisis classification.The authors cite published crisis-line guidance and a clinical-review process.Source: §8 · p. 11
• TWIST is vendor-proposed, and the authors require at least two external system runs before dropping “proposed” from its benchmark label.Public keys, outputs, harness, separability audit, and errata process are listed as structural bias mitigations.Source: §8 · p. 11

9 Conclusion

• TWIST treats retrieval correctness as necessary but insufficient, measuring whether systems intervene appropriately at belief change points and refrain when intervention is unwarranted.Its four tracks cover tension detection, draft alignment, belief supersession, and safe recall, each with matched false-intervention controls.Source: §9 · p. 11
• Track B’s structural gates, generator-failure taxonomy, and model-council pre-screen are presented as reusable methods for contradiction benchmarks.Tracks A and C and the CS-domain corpus are deferred to v2; Track D includes an opt-in safety corpus and clinical-review process.Source: §9 · p. 11
• The authors make independent competitor runs and public errata challenges requirements for TWIST’s credibility.They invite authors of cited systems and benchmarks to provide the first external runs.Source: §9 · pp. 11–12

10 Acknowedgment

• Marvin Danig is thanked for discussions that led to the work.Kinshuk Attri is thanked for helping with annotations.Source: §10 · p. 12

A Datasheet for TWIST (Track B v1)

• The Track B v1 datasheet describes 200 draft-vetting items from 10 public LoCoMo conversations, with synthetic personas and frozen held sets that are not for training.Items are LLM-generated from parameterized templates, passed structural and verification gates, and labeled by two gold-blind annotators with adjudication; κ and drop counts are reported in §5.Source: A Datasheet for TWIST (Track B v1) · p. 13

B Scenario family catalog

• Scenario families pair each detection family with surface-matched hard negatives; Track A and C and D families cover reversals, updates, sensitive probes, and controls.Track B’s instantiated table identifies hard-negative families as gold aligned=true; full templates ship with the repository.Source: B Scenario family catalog · p. 13, Table 4 · p. 14

C Prompts and judge decoy protocol

• The Track B verification prompt judges contradiction and explicit attribution from retrieved excerpts alone, treating them as the latest known record.Track C judge decoys test stale-as-current answers, perturbed details, fluent non-answers, and unordered histories; judges must reject at least 95% in every class for admissibility.Source: C Prompts and judge decoy protocol · p. 14

Opt-in distribution

• Crisis plants are excluded from default downloads and require an explicit flag plus confirmation; runs without them report Track D as “PII-only.”The file has a content-warning header and scenario-family tags, and the remaining tracks run without it.Source: D Crisis-content handling · p. 15

Guidance sources

• Plants use non-graphic, established safe-messaging phrasings, synthetic personas, and locale-aware referral gold.Guidance sources are enumerated in the release notes for auditability.Source: D Crisis-content handling · p. 15

Annotator safety

• Crisis items are isolated in a labeled workbook block annotators may decline without invalidating their other work.The protocol uses content notices, minimal excerpts, no time pressure, no surrounding-session rereading for adjudication, and provides regional crisis-line information.Source: D Crisis-content handling · p. 15