DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents

Motivation

• Memory hygiene matters because retrieval can surface stale or overgeneralized facts, while DreamBench-SWE tests superseded architecture, generated-file boundaries, scoped feedback, spurious failures, and abstention.A representative S3 task requires reproducing a reviewer-specified private dialect marker byte-for-byte when that marker is unavailable from the prompt and filesystem.Source: §1 · pp. 1–2

Reference probe

• The reference probe preserves raw trajectories while deriving typed memories, stale or superseded markings, retrieval blocks, and other repair artifacts for auditable comparison.Its variants include typed-only, raw-only, and hybrid configurations in a training-free wake/sleep/read architecture around a fixed software-engineering agent.Source: §1 · pp. 1–3

Research questions

• The paper’s research questions cover benchmark sensitivity, memory-system ordering, structured provenance-linked memory, executable hygiene diagnostics, and successor external-system auditing.The stated scope includes testing whether benchmark failures are hidden by single-session success and whether frozen cross-configuration mechanism contrasts remain evaluable.Source: §1 · p. 2

Original folds

• The scaled v2 primary comparison was null: the reference-probe hybrid achieved 95/180 = 0.528 versus B5’s 89/180 = 0.494, with permutation p=0.518 and Holm p=1.0.This is a failure to reject, not evidence of equivalence; C9 and C10 also exposed B0-headroom limitations for anti-hoarding claims.Source: §1 · p. 2

Benchmark

• DreamBench-SWE uses hand-curated traps requiring hidden, non-inferable earlier-session evidence and executable S3 oracles, with validity gating and contamination review.Wake agents and sleep judges exclude the benchmark repository, hidden oracles, reference solutions, and sequence records from their container filesystems, without claiming network isolation.Source: §1 · pp. 1–2

Contributions

• The successor audit separated concurrent memory-bearing controls and one admitted hosted literal-storage configuration from no memory, while its mechanism family was unavailable after conformance rejection.The resulting claim is benchmark discrimination plus one bounded external-system profile, not a mechanism, product ranking, superiority, or equivalence result.Source: §1 · p. 3

Evaluation gap

• DreamBench-SWE evaluates controlled repository continuation where hidden, non-inferable earlier-session evidence must drive code edits that pass executable oracles, under CSPRNG injection and scored-artifact isolation.Its C1–C10 taxonomy targets SWE-specific memory pathologies rather than generic issue resolution, dialogue recall, or agent-computer interaction [10] [37].Source: §2 · pp. 3–4

Agent memory architecture

• The reference probe repurposes episodic, semantic, and procedural memory vocabulary while adding training-free typed provenance, contradiction repair, grounded replay, stale suppression, and executable-oracle retrieval gating [28].This differs from virtual-context memory management, token-space curation, and offline context computation associated with MemGPT, Letta, and Sleep-time Compute.Source: §2 · p. 4

External memory systems and agent-native memory

• Mem0 is the only external memory-system family with scored conditions, while the successor audit admitted only B5-MEM0-LIT and treated rejected systems as conformance attrition rather than observed losses.The comparison distinguishes hosted extraction or literal storage from the reference probe’s offline typed maintenance; Supermemory provenance does not establish evaluated performance.Source: §2 · pp. 4–5, Table 2 · p. 6

SWE-agent memory and task-state substrates

• DreamBench-SWE differs from subtask-level SWE-agent memory, which reports a mean Pass@1 gain of +4.7 percentage points on SWE-bench Verified across four backbones.Its B6 condition is only an inspired substitute, while DreamBench-SWE tests hidden exact-token contracts, stale facts, contradictory feedback, and executable repository oracles.Source: §2 · pp. 4–5

Reflection, replay, and procedural lessons

• The paper treats reflection, replay, procedural lessons, durable task graphs, and memory-safety frameworks as adjacent foundations rather than novelty claims [22] [23] [32] [35] [42] [40] [38] [41].Its added distinction is lifecycle-managed, provenance-linked maintenance evaluated by executable S3 pass@1 and diagnostics for stale, contradictory, harmful, and repeated-error behavior.Source: §2 · p. 5

Research and knowledge-curation agents

• Research and retrieval-synthesis agents broaden or curate external knowledge, whereas DreamBench-SWE evaluates maintenance of a coding agent’s own prior software-work history [9] [24] [18] [19] [31] [33].This boundary complements the benchmark’s refusal to claim generic priority for multi-session, execution-oriented, or lifecycle-aware memory evaluation.Source: §2 · pp. 3, 5

Core entities

• A session is a bounded agent run, and a sequence is an ordered list of sessions over one repository or related repositories.Trajectories record immutable evidence, metadata, events, outcomes, final diffs, verification status, cost, latency, and labels.Source: §3 · pp. 6–7

Memory representation

• Memory items carry typed content, provenance, scope, lifecycle status, validity, confidence, utility, risk, staleness, and contradiction or supersession links.Logical deletion blocks retrieval while preserving derived-memory records and provenance; raw trajectory evidence remains intact in reference-probe experiments.Source: §3 · p. 7

Read admissibility

• Read admissibility separates candidate generation from admission, rejecting memories whose status, validity, scope, supersession, risk, staleness, provenance, or type policy is unsuitable.Admissible memories are ranked under a fixed memory-token budget using signals such as semantic similarity, tags, paths, symbols, tasks, and project scope.Source: §3 · p. 7

Hygiene objective and metrics

• Memory hygiene evaluates useful, grounded, current, scoped, low-risk retrieval while suppressing harmful or stale items, rather than maximizing generic recall.The confirmatory fold logged HarmfulMemoryRate as 0.000 for every condition, so it does not support harmful-retrieval reduction claims; stale activation was diagnostic rather than confirmatory.Source: §3 · p. 7

Research question

• RQ0 compares offline maintenance, raw-evidence retention, and their combination under fixed model, token, tool, and task-stream controls.Descriptive v2 ablations ranged from episodic-only 20/180 = 0.111 to no contradiction repair 88/180 = 0.489, leaving contradiction-repair and operator-specific causal claims as hypotheses.Source: §3 · p. 8

4 Method: Reference Probe

• The probe wraps a fixed wake-phase software-engineering agent with an external memory store and offline maintenance loop, without training a new base model or exposing internal state.The wake agent may be a SWE-agent-style harness or another code-capable tool-using agent.Source: §4 · p. 8

4.1 Wake/Sleep/Read Loop

• Each session initializes a repository snapshot and condition-specific namespace, retrieves gated memories, runs the wake agent, logs execution evidence, and optionally performs offline maintenance.The logger records prompts, configurations, tool calls, outputs, diffs, memory operations, feedback, outcomes, cost, and latency.Source: §4.1 · p. 8

M Derived store append records soft-delete status

• Raw episodes remain append-only and fully addressable, while sleep operators may append derived memories, audit records, or lifecycle changes but cannot rewrite or delete raw evidence.Logical soft-deletion applies only to derived memories, preserving provenance and replayability.Source: §4.1 · p. 8, Figure 1 · p. 9

E Raw episode log

• The evaluated ladder uses frozen repairs for verbatim raw-evidence capsules and contradiction exclusion, defining typed-only, raw-only, and hybrid reference-probe variants.Raw-only is descriptively near-equivalent to B5 on recall-verbatim evidence, with 65/66 historical paired S3 outcomes agreeing; this is not an equivalence test.Source: §4.1 · p. 8, E Raw episode log · p. 9

4.2 Typed Memory Schema

• Typed memories require content, type, provenance, and write reason, with scope, lifecycle, confidence, utility, risk, and staleness metadata governing maintenance and retrieval.The schema defines eight memory types, while insufficiently grounded records become requires_review and are ineligible when provenance gating is enabled.Source: §4.2 · pp. 9–10

4.3 Sleep-Phase Operators

• Sleep-time operations are named ablations, but the confirmatory fold isolates only typed, raw-evidence, and hybrid ladder channels rather than statistically testing every component.A0 episodic-only is a descriptive floor, and implemented operators are separated from fold-isolated ablations.Source: §4.3 · p. 10, Table 3 · p. 10

4.4 Retrieval Gate

• Retrieval first applies hard scope, validity, provenance, risk, staleness, lifecycle, and condition-specific type gates, then ranks admitted memories with metadata-aware scoring.Weights and thresholds are frozen before held-out evaluation; the similarity-only comparator was not run, and several gate ablations remained designed-not-run.Source: §4.4 · p. 11

4.5 Condition Isolation

• Separate memory namespaces and per-call containerized filesystem isolation prevent cross-condition contamination and keep benchmark artifacts, solutions, records, and analysis files unavailable to agents.The run is not network-isolated, and Section 5.6 describes the associated adversarial audit and protocol.Source: §4.5 · pp. 11–12

5.1 DreamBench-SWE

• DreamBench-SWE uses 60 admitted three-session traps in which S3 requires repository-specific use of non-inferable evidence from earlier sessions without hidden-oracle or reference-solution access.The benchmark is a curated diagnostic stress benchmark rather than broad coverage of production issue resolution; 18 of 22 historical v1 traps were recall-verbatim, while all traps still required implementation.Source: §5.1 · p. 12

5.2 Conditions

• B5 is a deterministic verbatim event-memory substitute, while the reference probe and hosted Mem0 configurations are explicitly non-fidelity comparisons to third-party systems.B5 writes one record per trajectory; the pinned hosted configurations use mem0ai 2.0.11, top_k=6, and a 1,200-token memory-read budget.Source: §5.2 · pp. 13, 15

5.3 Controls

• The v2.1 audit froze conformance gates, budgets, comparison families, unavailable-slot handling, and analysis before outcome inspection, using trap-clustered exact sign/permutation tests.Four conditions cleared conformance; three others remained registered unavailable slots rather than observed performance rows, and Holm families were not shrunk.Source: §5.3 · pp. 15–16
• The canonical fold narrowly missed the preregistered B0 warmup threshold, so its S3 failures remain reportable but are not interpreted uniformly as memory-only failures.B0 warmup was 287/360 = 0.797 against the required 0.80 threshold.Source: §5.3 · p. 16

5.4 Metrics

• The study reports Pass@1, final_passed, and eight memory-hygiene metrics, while collapsing numerically identical judge-artifact metrics and treating HarmfulMemoryRate and RegressionAfterUpdate as diagnostic.Lower is better for RepeatedErrorRate, HarmfulMemoryRate, and RegressionAfterUpdate; higher is better for the remaining hygiene metrics.Source: §5.4 · p. 17

5.5 Run Protocol

• Frozen records support exact replay of the checked-in fold, whereas live reruns using hosted gpt-5.5 can drift and therefore constitute new experiments.The protocol fixes prompts, commands, model configurations, budgets, conditions, and seeds, and keeps hidden oracle answers and reference solutions out of agent containers and public prompts.Source: §5.5 · p. 17

5.6 Isolation and Contamination Control

• The benchmark guarantees filesystem isolation from scored artifacts but not network isolation, and reports no evidence of hidden benchmark-file reads without claiming a zero contamination diagnostic.The container path is fail-closed, and the isolation acceptance suite passed 3/3 tests.Source: §5.6 · pp. 18–19

6.1 V2 Baseline Sweep and Construct Strata

• 89/180 = 0.494 for B5, 95/180 = 0.528 for the hybrid reference probe, 80/180 = 0.444 for typed-only, and 84/180 = 0.467 for raw-only defined the descriptive v2 ladder.These pooled rates were descriptive, with hosted live-memory baselines excluded from the headline table.Source: §6.1 · p. 19, Table 7 · p. 20

6.2 Clustered P-Family

• 95/180 versus 89/180, the v2 reference-probe hybrid did not significantly separate from B5 under the preregistered clustered P-family.The complete comparison had 60 traps, three seeds, 180 paired cells, no missing cells, and zero excluded records; failure to reject was not equivalence.Source: §6.2 · p. 19, Table 9 · p. 21

6.3 Admission Funnel and Construct Validity

• C9 and C10 failed the strengthened B0-headroom criterion because B0 passed 12/12 and 6/6 cells, respectively, invalidating them as anti-hoarding or abstention evidence in this fold.After excluding these strata, 24 traps across C2, C3, C5, C6, and C7 satisfied the preregistered headroom criterion, but this did not establish construct validity.Source: §6.3 · p. 20, Table 10 · p. 21

6.4 Clean-start and Hygiene Diagnostics

• 1434/3024 = 0.474 IrrelevantImportRate, 378/2268 = 0.167 OverscopeRate, 1512/2646 = 0.571 RepeatedErrorRate, and 756/3780 = 0.200 StaleUseRate diagnosed memory-failure modes rather than system wins.The clean-start subset was descriptive and non-claim-rescuing: the hybrid reached 19/39, B5 15/39, and B3 22/39.Source: §6.4 · pp. 20–21, Table 11 · p. 21

6.5 Probe Ablations and Raw-evidence Checks

• The raw-evidence invariant is an auditability and measurement property, not a proven causal mechanism: raw episodes remain available for inspection, while derived memories can be marked stale or gated.Source: §6.5 · p. 22

6.6 Separately Preregistered v2.1 External-Systems Audit

• All three available Family A comparisons against B0 rejected after Holm correction, whereas rejected external configurations remained unavailable rather than observed losses.B5, DF-hybrid, and B5-MEM0-LIT each separated from no external memory; Family B had no evaluable mechanism contrast.Source: §6.6 · pp. 22–23, Table 13 · p. 23
• 97/180 versus 82/180, B5-MEM0-LIT’s secondary advantage over B5 was nonconfirmatory and sensitivity-dependent, while 97/180 versus 83/180 did not reject against DF-hybrid.The successor therefore supports benchmark discrimination and characterizes one exact hosted literal-storage configuration, without establishing superiority among memory-bearing conditions.Source: §6.6 · p. 23

Primary comparison

• 95/180 versus 89/180 favored DF-hybrid numerically over B5, but clustered p = 0.518 and Holm p = 1.0 yielded no rejection.This is a failure to reject, not evidence of equivalence or practical ties.Source: §7 · pp. 23–24

Family-level reading

• All six preregistered v2 clustered comparisons were non-rejections after Holm correction, including DF 80/180 versus B3 67/180 with Holm p = 1.0.Typed-only and raw-only also failed to beat B5, so the system story remains diagnostic and mechanistic rather than confirmatory.Source: §7 · p. 24

Benchmark validity

• 30 traps passed validation and admission, 1890/1890 condition-level result files were produced, and B0 reached 21/180 while strong baselines separated on C2, C3, C5, C6, and C7.C9 and C10 were invalid as anti-hoarding or abstention evidence because B0 passed every cell, although C2, C3, C5, C6, and C7 retained 24 traps.Source: §7 · p. 24

Hygiene diagnostics

• DF-hybrid reduced RepeatedErrorRate to 0.142 versus 0.225 for B5, but UsefulMemoryPrecision was 0.264 versus 0.556 and RegressionAfterUpdate remained 0.106.These are error-taxonomy diagnostics, not evidence that the maintenance pipeline solved memory hygiene.Source: §7 · p. 24

Supplemental diagnostics

• B5-MEM0 reached 21/180 and B5-MEM0-LIT 20/180, while raw-only, hybrid, and B5 cost 0.03556, 0.03832, and 0.03953 USD per successful task, respectively.These results characterize pinned configurations and diagnostic costs; they do not rank tuned systems or rescue the null task comparison.Source: §7 · pp. 24–25

Successor external-audit reading

• The v2.1 audit found concurrent controls and one admitted hosted literal-storage configuration separated from B0 after Holm correction, while mechanism contrasts were unavailable after conformance rejection.The literal configuration’s comparison with B5 was sensitivity-dependent, its comparison with DF-hybrid did not reject, and the result does not establish external-system superiority or causal attribution.Source: §7 · p. 25

• 95/180 versus 89/180 is a null DF-hybrid–B5 result, not evidence of equivalence or a hidden win.A future superiority or equivalence claim would require a new preregistration with suitable power and equivalence margins.Source: §8 · p. 25
• B0 misses the warmup gate at 287/360 = 0.797, while C9 and C10 lack no-memory headroom, constraining anti-hoarding and abstention interpretations.The benchmark still exposes differentiated failure modes, but these strata are not uniformly validated in this fold.Source: §8 · p. 25
• DF-hybrid’s UsefulMemoryPrecision is 0.264 versus B5’s 0.556, showing that this typed retrieval implementation did not improve retrieved-memory usefulness.This limitation is independent of the null primary comparison; hygiene labels remain diagnostic, task-coupled, and not independent proof of improved task success.Source: §8 · p. 26
• Only one wake-model family and one hosted service route were fully evaluated, so no broad model-generality or deployment claim is justified.GLM-5.2 did not clear warmup feasibility; hosted behavior can drift, and the successor does not generalize to other wake models, retrieval policies, deployments, service versions, or production repositories.Source: §8 · pp. 26–27
• DreamBench-SWE uses controlled fixtures and filesystem isolation rather than a hermetic network-isolated environment, limiting ecological validity and the scope of its security guarantee.Its hidden-evidence repository-continuation claim is distinct from generic priority over neighboring benchmarks or transfer to open-ended production SWE; “sleep” and “dream” refer only to offline computation and maintenance artifacts.Source: §8 · pp. 26–27
• The study evaluates bounded memory implementations and one admitted hosted successor configuration, so its results do not rank tuned Mem0, Mem0g, other products, or unavailable successor conditions.B5 is a deterministic verbatim substitute, while B5-MEM0 and B5-MEM0-LIT are pinned configurations; native B5-MEM0 and both Supermemory conditions produced no successor performance result.Source: §8 · pp. 26–27

Benchmark contribution

• DreamBench-SWE combines controlled repositories, hidden executable oracles, frozen sequence records, injected non-inferable facts, validity gates, and trap-clustered inference for multi-session memory-hygiene evaluation.The reference probe combines raw trajectory retention with typed, provenance-linked memory maintenance.Source: §9 · p. 27

v2 confirmatory fold

• 95/180 versus 89/180, p = 0.5184192657470703, and Holm-adjusted p = 1.0 yielded no rejection for DF-hybrid versus B5 in the v2 fold.This bounded negative result establishes neither superiority nor equivalence over strong verbatim event memory.Source: §9 · p. 28

Construct limitations

• C9 failed stratum validity because B0 passed all cells, while C10 lacked B0 headroom, preventing spurious-lesson-rejection and abstention claims in the fold.These caveats narrow rather than invalidate the benchmark artifact.Source: §9 · p. 28

v2.1 successor audit

• 21/180, 82/180, 83/180, and 97/180 passes were recorded for B0, B5, DF-hybrid, and B5-MEM0-LIT; Family A comparisons against B0 rejected after Holm correction.Both Family B mechanism contrasts were unavailable after pre-evaluation conformance rejection, and the successor did not establish superiority among memory-bearing conditions or broad product generality.Source: §9 · p. 28

Scope of evidence

• The release justifies transparent benchmark artifacts and an additive external-system profile, but not model generality, generic benchmark priority, tuned-Mem0 ranking, production reliability, or favorable cost claims.The evidence comes from one wake model family and one seed schedule.Source: §9 · p. 28

Definition and interpretation

• Recall-verbatim means the prior injected event text contains the S3 contract later checked by the hidden oracle, not that answer code or a complete patch can be copied.The classification is semantic rather than based on crude token-overlap code-diff fields.Source: A Recall-Verbatim Trap Audit · pp. 32–33

Per-trap classification

• 18 of 22 v1.0 S3 traps are recall-verbatim, while four are synthesis/application traps requiring generator workflow beyond a partial token.Table 14 reports 18 recall-verbatim rows and four synthesis/apply rows; all 22 traps still require implementation, and a semantic audit found zero pure copy-and-stop traps.Source: A Recall-Verbatim Trap Audit · pp. 32–33, Table 14 · p. 33

Validity qualification

• 44/65 remains the validity-gated reference-probe typed-only denominator, because one displayed non-success was excluded after codex_exec_failed.The displayed outcome cells are taxonomy counts across seeds 1–3, not the validity-gated denominator.Source: A Recall-Verbatim Trap Audit · p. 33

Limitations

• Even recall-verbatim traps can fail: config-reviewer-coerce-tag and expr-stale-registry-stability still require sufficient implementation work after the contract is available.Thus, verbatim availability does not make B5 success automatic.Source: A Recall-Verbatim Trap Audit · p. 33

B Artifact and Reproducibility Details

• The public package reproduces v2 summaries from the canonical analyzer bundle without live hosted-model reruns, with completion and freshness checks reporting STATUS PASS failures=0.The v2 fold uses a frozen 60-trap sequence over seeds 1, 2, 3, with gpt-5.5 as wake model and codex-gpt-5.5 as judge; supplemental hosted-Mem0 rows are excluded from the P1–P6 family.Source: B Artifact and Reproducibility Details · pp. 33–34

B.1 v2.1 Successor Evidence Identity

• The v2.1 successor is an additive release distinct from immutable v2.0.5, with pinned run, commit, launcher, lock, analyzer, and generated-surface digests while excluding private ledgers, credentials, hidden oracles, and local paths.The manuscript consumes generated macro surfaces through analysis/fold/v21_external_audit_macros.tex, with fail-closed provenance checks requiring exact copies in both manuscript trees.Source: §B.1 · p. 37

C V2 Benchmark Protocol Preregistration

• The V2 confirmatory protocol was frozen before trap authoring and performance runs, defines clustered validity-gated S3 pass/fail as primary, and explicitly scopes out the second wake-model transfer fold.V2 is a new scaled fold rather than a rewrite of V1; later changes require an amendment log.Source: C V2 Benchmark Protocol Preregistration · p. 37

C.1 Causal History and Bias Controls

• V2 controls prior-result bias through construct quotas, separate family labels, frozen candidate hashes and validation reports, CSPRNG secrets, template caps, admission-funnel disclosure, and mandatory publication of null primary results.These controls address risks including punishing B5’s strategy, emphasizing strong reference-probe constructs, tuning literals, or removing traps after early outcomes.Source: §C.1 · p. 38

C.2 Trap Set and Construct Quotas

• The frozen V2 target contains 60 traps and 180 S3 cells per complete condition, allocated by constructs C1–C10 with anti-hoarding floors, a separate family label, and at least 10 clean-start traps.The design uses 22 V1 traps, 8 synthesis-pilot traps, and approximately 30 new traps; clean-start S3 begins from a frozen clean fixture base and is reported separately.Source: §C.2 · pp. 38–39

C.3 CSPRNG Secret Injection and Validity Gates

• The harness injects cryptographic secrets and rejects traps lacking hidden, non-inferable, construct-appropriate S3 evidence, while scoring only permitted production diffs from continuation or clean-start states.For non-C1 traps, no single injected event may contain the decisive oracle literal or all required facts; construct-specific oracles also test prohibited behavior where required.Source: §C.3 · pp. 39–40

C.4 Authoring, Admission, and Audit

• All 30 authored new traps passed dry and live validation and were admitted, yielding a 60-trap V2 fold subject to skeleton-diversity limits and complete human construct-label review.At least 30% of admitted traps receive deep audit for fit, realism, leakage, duplication, and mechanism load-bearing; unresolved label disagreements cannot support per-construct claims.Source: §C.4 · pp. 40–41
• The second wake-model fold was scoped out, preventing the planned cross-backbone author-model × wake-model interaction analysis and any construction-independent validity claim.The planned secondary analyses included stratified pass rates, transfer performance, same-family author/solver advantage, and a possible sensitivity table; they could not replace the primary clustered comparison.Source: §C.4 · p. 41

C.5 Conditions and Run Freeze

• The primary fold fixed Codex CLI with gpt-5.5 for waking, codex-gpt-5.5 for judging where applicable, 60 valid traps, seeds 1, 2, and 3, task order, budgets, stopping criteria, and isolated memory namespaces.Hidden oracles, reference solutions, sequence records, analysis files, logs, and secret manifests were absent from the wake-agent filesystem.Source: §C.5 · p. 41
• The matrix included B0–B7, B5-MEM0, B5-MEM0-LIT, DF variants, and A-family conditions; B5 was deterministic verbatim offline memory, while the two Mem0 variants were pinned supplemental hosted configurations.B5-MEM0 and B5-MEM0-LIT entered only when package, service, namespace isolation, payload format, and key requirements were available, and they were excluded from the primary Holm family.Source: §C.5 · pp. 41–42
• Before the first primary-fold run, the preregistration, manifests, condition definitions, model and provider settings, decoding parameters, budgets, seeds, prompts, scorer, generator code, and secrets were frozen and hashed.Result roots, grid logs, Mem0 namespaces, and fixture/cache artifacts were isolated through dedicated environment variables.Source: §C.5 · p. 42

C.7 Second Wake-Model Fold (Scoped Out)

• No full second wake-model transfer fold is reported because GLM-5.2 failed the feasibility gate, so the paper makes no cross-backbone or transfer claim.The planned fallback chain and transfer fold were scoped out rather than pooled with or used to rescue the primary fold.Source: §C.7 · p. 43

C.8 Falsifiers, Go/No-Go, and Null Handling

• A null P1 result must be published as such: it does not establish DF-hybrid superiority over B5, and secondary analyses cannot become the headline.Falsifiers narrow or remove claims rather than authorize hiding the result; a losing system remains publishable if the benchmark measures its intended construct.Source: §C.8 · pp. 44–45

C.9 Required V2 Artifacts and Claim Boundaries

• V2 claims are bounded by frozen analyses: primary system claims require clustered Holm-corrected P1 success, while transfer, clean-start, typed-maintenance, raw-evidence, and construct claims require their corresponding evidence.Failure to reject cannot be presented as equivalence, and no secondary analysis can replace clustered P1.Source: §C.9 · p. 46

D v2 Construct Taxonomy

• The v2 taxonomy assigns C1–C10 to distinct memory-hygiene failure mechanisms, replacing legacy family labels as the basis for confirmatory construct quotas.C1 covers exact recall, whereas later constructs test interference, staleness, propagation, scope, authority, composition, procedural source-of-truth, disconfirmation, and abstention.Source: D v2 Construct Taxonomy · p. 46, Table 21 · p. 47

E Clustered Statistical Methods; E.1 Trap-as-Cluster Model; E.4 Why Pooled McNemar Over-Counts Evidence

• The primary statistical cluster is the trap: seeds are repeated observations, and exact label swaps occur jointly across each trap’s valid paired seeds.This prevents the three seeds sharing one hidden contract, repository pathology, oracle, and implementation burden from being treated as independent evidence.Source: §E.1 · p. 48, §E.2 · p. 48, §E.4 · p. 49

C.6 Primary Analysis; E.1 Trap-as-Cluster Model; E.2 Exact Trap-Clustered Sign/Permutation Test

• Dobs = +6 for DF-hybrid versus B5, with 21 versus 15 discordant cells, yielded clustered p = 0.5184192657470703 and Holm-adjusted p = 1.0.The 60 traps, not 180 seed-trap cells, are the primary exchangeable units; tied cells contribute zero and no normal approximation is used.Source: §C.6 · p. 43, §E.1 · p. 48, §E.2 · pp. 48–49

E.3 Successor Families and Unavailable Slots

• The v2.1 successor audit keeps unavailable comparisons in their registered families with raw and adjusted p = 1, while literal-storage comparisons remain secondary and nonconfirmatory.Holm correction is applied separately to the six-slot benchmark family and two-slot mechanism family; conformance rejection does not become an observed loss.Source: §E.3 · p. 49

E.4 Why Pooled McNemar Over-Counts Evidence; C.6 Primary Analysis

• Pooled McNemar is only a secondary cell-level sensitivity check because treating 66 or 180 seed-trap cells as exchangeable over-counts evidence within 22 or 60 correlated trap clusters.The clustered test instead assigns one exchangeable sign per trap while retaining the trap-level directional magnitude.Source: §C.6 · p. 43, §E.4 · p. 49

E.5 v1 Clustered Sensitivity Result

• Among 65 paired cells, reference-probe typed-only (DF) versus B3 reflection-only vector retrieval was the surviving comparison, with b = 24, c = 3, and exact p = 4.92334365845 × 10−5.The reference-probe typed-only condition was missing seed3:expr-s tale-registry-stability.Source: §E.5 · p. 49
• The same comparison remained significant under the trap-clustered exact test, with observed b −c = 21, raw clustered p = 0.00732421875, and Holm-adjusted p = 0.03662109375.The test used 22 contributing trap clusters and 65 paired seed cells.Source: §E.5 · p. 50
• The v1 finding was narrow: all three per-seed tests failed to reject after seed-family Holm correction, and the three-seed trap-majority collapse had Holm-adjusted p = 0.1953125.The paper-level interpretation is a positive structured-memory comparison under the exact trap-clustered test, not adequate pooled inference, with seed-concentration and trap-collapse caveats.Source: §E.5 · p. 50

E.6 Power and Minimum Detectable Effects at 180 Cells

• The v2 design’s primary power is determined by the number of nonzero-discordance traps and within-trap seed clustering, rather than 180 independent cells.With k equal-weight nonzero traps, the trap-clustered test reduces to an exact sign test.Source: §E.6 · p. 50
• 39–21 is the smallest equal-weight split below α = 0.05, whereas 38–22 is not below 0.05 in the full k = 60 case.These thresholds use the frozen 180-cell denominator only for descriptive cell margins.Source: §E.6 · p. 50, Table 22 · p. 50
• The same trap-level rejection boundary can represent either a 10 percentage point or 30 percentage point cell-level net margin, depending on within-trap seed clustering.With one discordant seed per trap, the descriptive margin is smaller than when all three seeds in each trap favor the same direction.Source: §E.6 · pp. 50–51
• The design has reasonable power only for a large, stable trap-level reference-probe advantage and may miss small effects, sparse discordance, or effects concentrated in a few trap templates.The reported fold computes observed dt vectors and primary p-values in analysis/fold/v2_confirmatory_clustered.json.Source: §E.6 · p. 51, Table 23 · p. 51

F Reproducibility Appendix

• The runbook separates paper sources, benchmark code, frozen records, rerun machinery, validation scripts, hidden scoring assets, and result roots into distinct repository areas.The layout identifies paper/, src/, scripts/, experiments/env/, experiments/results/, analysis/, artifact/, and dist/ as separate reproducibility components.Source: F Reproducibility Appendix · p. 51
• Exact replay depends on frozen sequence commits, seeds, hidden assets, harness code, container configuration, and result metadata, whereas rerunning the launcher creates a new hosted-model experiment.The launcher is explicitly a corrected replay-ready runbook rather than a reconstruction of the historical fold, whose completeness comes from the completion audit and folded records.Source: F Reproducibility Appendix · pp. 52, 54, 55
• Pinned hosted-Mem0 failure analysis is supplemental: its historical 6/66 S3 pass rate reflected exact-token corruption, including replacement of an ASCII hyphen with a Unicode non-breaking hyphen.The appendix states that this diagnoses one pinned configuration and is not a practical ranking claim; the v2 diagnostic configurations remained near B0 at 21/180 and 20/180.Source: F Reproducibility Appendix · p. 55
• 21/180 B0 passes versus 82/180 B5, 83/180 DF-hybrid, and 97/180 B5-MEM0-LIT passes, with all three available comparisons against B0 surviving Holm correction.The analyzer-bound surface reports rates of 0.116667, 0.455556, 0.461111, and 0.538889, respectively; B5-MEM0, B5-SM-LOCAL, and B5-SM-LOCAL-DOC comparisons were unavailable.Source: F Reproducibility Appendix · p. 55

G Notation and Glossary

• The notation and glossary appendix introduces no new empirical claims and defines the paper’s notation, condition codes, session and aggregation terms, repairs, operators, metrics, and statistical terminology.Its definitions draw from the paper sections, construct taxonomy, artifact appendix, analyzer, and scoring helpers; construct profiles remain descriptive unless tied to a preregistered comparison.Source: G Notation and Glossary · p. 56

G.1 Core Notation

• The core notation tables define the problem formulation, method notation, and retrieval-gate terms used by the paper.These definitions are organized in Tables 24 and 25.Source: Table 24 · p. 57, Table 25 · p. 58

G.2 Sessions, Outcomes, and Aggregation

• The glossary specifies session, outcome, aggregation, condition-code, repair, sleep/read, and construct-label terminology, including designed-but-not-run ablation labels.Tables 26–30 organize these definitions, with construct-profile outcomes reported separately by canonical analyzer artifacts.Source: Table 26 · p. 59, Table 27 · p. 60, Table 28 · p. 60

G.6 Metrics and Hygiene Terms

• Hygiene claims are diagnostic because several metrics are task-coupled, collinear, or judge-dependent, with confirmatory-fold values deterministically re-scored offline.Source: Table 31 · p. 63
• RepeatedError Rate counts repeated labeled errors recurring in an active patch, divided by opportunities to avoid repetition.Source: §G.6 · p. 63
• StaleMemory Activation Rate measures stale or superseded memories used among tasks where stale memory was available, while Scope Accuracy aggregates scope-matched retrieval behavior.Source: §G.6 · p. 63
• RegressionAfter Update and Contradiction Repair Accuracy are diagnostic measures, with the latter requiring correct old-item status, new-item status, and relation.Source: §G.6 · p. 63
• HumanFeedback Use Accuracy measures useful in-scope feedback retrieved and reflected in the patch, but it is collinear with CRA and TransferScore in ladder conditions.Source: §G.6 · p. 63

G.7 Clustered and Paired-Test Terms

• Family A is the fixed six-slot memory-versus-B0 discrimination family, whereas Family B is the fixed two-slot cross-configuration mechanism family.Source: §G.7 · p. 65
• A conformance-unavailable slot is rejected before evaluation, remains in its Holm family at p = 1, and is neither an observed null nor an observed loss.Source: §G.7 · p. 65
• Secondary comparisons are unprotected for confirmatory superiority claims and require trap-majority and per-seed sensitivity checks, while temporal-drift bands are diagnostic tolerances rather than equivalence margins.Source: §G.7 · p. 65

Benchmark-contamination resistance

• Fresh CSPRNG tokens keep decisive hidden literals out of public artifacts, so visible repositories and released benchmark materials cannot reveal token-dependent answers except by guessing.This protects against public training-set leakage but does not establish network isolation or prevent contamination from accidentally mounted hidden files.Source: H Ethics, Broader Impact, and… · p. 66

Dual-use scope

• The benchmark measures software-agent memory hygiene rather than exploitation, malware, credential theft, surveillance, or social engineering, and uses synthetic repositories, events, and tokens without personal or production data.Its principal privacy risk is accidental leakage of hidden answers or local operational material, addressed through packaging and isolation checks.Source: H Ethics, Broader Impact, and… · p. 66

Compute cost and environmental honesty

• 0.03832 USD per successful task is the estimated reference-probe hybrid cost, versus 0.03953 for B5 and 0.03556 for reference-probe raw-only; these are diagnostic figures, not a favorable cost claim.The primary task comparison remains null.Source: H Ethics, Broader Impact, and… · p. 66
• Carbon emissions and energy consumption are not reported because hosted endpoints lack verified power, utilization, energy-mix, and per-request accounting; sleep-phase maintenance increases model-call volume and limits practical claims.Future deployment evaluations should report token budgets, call counts, runtime, and provider-backed energy or emissions estimates where available.Source: H Ethics, Broader Impact, and… · pp. 66–67

Hosted-Mem0 limitation

• Pinned hosted Mem0 configurations are failure-analysis cases, not a ranking of Mem0: audited retrieval was followed by exact-token corruption under the reported harness.The result does not generalize to tuned, self-hosted, differently configured, or explicitly verbatim-preserving variants; the successor audit preserves these disclosure boundaries.Source: H Ethics, Broader Impact, and… · p. 67