PaperlayerView exact revision on arXiv

Source-linked AI summary

When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory

Kazuki Nakayashiki

arXiv:2608.25553v1cs.IRcs.AIcs.CL

TL;DR

Inherited memory can preserve a constraint after a newer authoritative record withdraws it, raising the question of whether scarce verification recovers the change. The paper models immutable provenance and compares native allocation with same-budget reassignment to the critical path, finding large gains in current-record-consistent decisions while clarifying that the oracle intervention is not deployable.

  • Problem

    The paper examines whether agents recover withdrawn constraints in inherited memory when verification is scarce.

  • Method

    The study models supersession with immutable provenance and assigns memory form, world state, and verification policy at a fixed budget of two records.

  • Results

    +74.0, +72.7, and +61.3 points in current-record-consistent decisions followed critical-path reassignment across the primary, replication, and held-out runs.

  • Takeaways & Limitations

    Verification allocation can determine whether stale inherited state is corrected, motivating freshness or supersession signals separate from relevance.

  • Takeaways & Limitations

    The forced-critical intervention uses experimenter knowledge of the critical path, so it identifies avoidable error rather than providing a deployable scheduler.

Abstract

from arXiv · show

An agent that inherits a consolidated memory may inherit a constraint that was true when written and has since been withdrawn by a newer authoritative record. Under a scarce verification budget, does the agent recover the withdrawal, and if not, is the error avoidable without spending more? We model supersession explicitly -- historical provenance is immutable; what changes is which record is current -- and assign by design the memory's form, the world's state (source current or superseded), and the verification policy at a fixed budget of two records: the agent's own allocation, or the same budget with one slot re-assigned to the critical provenance path or to a random record. With a constraint stated, agents inspected its provenance path in about one episode in five; when that constraint had been superseded, native allocation produced stale-consistent decisions in 77.3%, 74.7% and 74.7% of episodes across a primary run, a fresh-wording replication and a held-out domain. Re-assigning one slot to the critical path raised current-record-consistent decisions by +74.0, +72.7 and +61.3 points, positive in six of six models in each of those runs, and changed nothing when the record agreed with the memory. The held-out scenario was later found to contain a temporal inconsistency; a robustness replication with one sentence corrected, deposited externally before execution, gave +73.3 points and is reported alongside the original. The intervention uses knowledge of the critical path and is not a scheduler; it identifies that the share of stale-memory error attributable to verification allocation is close to its structural ceiling. Memory systems may need freshness or supersession signals separate from relevance.

1 Introduction

The paper asks whether agents can recover withdrawn constraints in inherited memory when verification is scarce, and whether reallocating the same budget avoids the resulting failure. It introduces temporal supersession while treating provenance as immutable and tests native versus targeted verification.

  • Research question: The paper tests whether budgeted verification creates avoidable failures when a previously true stated constraint becomes stale.This extends prior work that measured where agents allocate checks while all constraints remained true.
  • Research question: A memory is consolidated from source record S0 at t0, then S1 supersedes S0 and withdraws the constraint at t1 without re-consolidation.Historical provenance remains unchanged, while the current record changes.
  • Design: The study compares native allocation with policies that reassign one of two verification slots to the critical provenance path or a random non-critical record.The verification budget remains fixed at two records.
  • Findings: 77.3%, 74.7%, and 74.7% of superseded-world episodes followed the stale memory rather than the current record under native allocation.These values come from the primary, replication, and held-out procurement runs.
  • Findings: +74.0, +72.7, and +61.3 points in current-record-consistent decisions followed critical-path reassignment across the three runs.The gains were positive in six of six models in each run and decisions changed little when the record agreed with memory.
  • Scope: The intervention identifies the allocation-attributable share of stale-memory error but uses experimenter knowledge of the critical path and is not a deployable scheduler.The paper does not identify why native allocation selects its records or measure stale stated constraints in deployed stores.

2 Provenance, supersession, and stale memory

The provenance model separates immutable historical links from changing record currency. A memory is stale only when its current provenance target withdraws the content it states, and verification can reveal that supersession through one archive request.

  • Objects: Historical provenance records how memory M was derived from S0, while supersession appends that newer record S1 became authoritative and S0 is no longer current.The provenance link remains immutable; supersession is append-only.
  • Objects: The current record is S1 when supersession exists and otherwise remains S0.Current currency is distinct from the historical provenance target.
  • Stale memory: A memory is stale when its current provenance record withdraws the content the memory states, not when the provenance link changes.The implementation never rewrites the provenance record.
  • Verification: The archive returns S0, its status, and S1 beneath it when S0 has been superseded.Because every record carries a status line, discovering the correction requires spending a verification request on that provenance path.
  • Timeline: At allocation time, the valid and superseded worlds present the agent with identical store lines.The worlds differ only in whether a later t1 supersession event occurred.

3 Experimental design

The experiment assigns memory form, world state, and verification policy while holding the verification budget at two source records. It measures whether the final action matches the record marked current.

  • Experimental design: The six-memory scenario gives agents a declining-metrics situation, five candidate actions, and a verification budget of k = 2 source records.The target memory concerns a targeted discount whose source record reports retention loss and a prohibition on reuse.
  • Experimental design: Memory form is stated or removed, while world state is valid or superseded and invisible at allocation time.The stated form includes positive evidence, a quantified negative outcome, and a prohibition.
  • Verification policies: Native verification returns the agent’s own two records; forced-critical returns the target path plus another agent-named record; forced-noncritical returns a seeded random non-target record plus that record.All policies operate within the same two-record budget.
  • Episode: The agent names up to two memory ids and a provisional action before receiving the archive records, status lines, and escalated situation.Both turns use strict JSON schemas and store prompts, responses, and scores.
  • Estimand: The headline estimand is the risk difference in current-record-consistent action Y between forced-critical and native within stated × superseded, pooled with equal model weights.The estimate uses a model-stratified bootstrap 95% interval with B = 4,000.
  • Runs: The primary run contains 2 × 2 × 3 = 12 cells, six models, and 25 episodes per cell, totaling 1,800 episodes.The replication also has 1,800 episodes, while the held-out run has 900.

4 Results

Native allocation rarely checks the stated constraint’s provenance, and superseded constraints consequently produce stale-consistent decisions. Reassigning one existing verification slot to the critical path removes most of that error across primary, replication, and held-out runs.

  • 4.1 Native allocation rarely selects the stated constraint’s path: 20.1% and 23.1% of stated-constraint episodes named the target provenance path in the primary and replication runs.With the constraint removed, naming rates were 66.9% and 72.9%, differences of +46.8 and +49.8 points.
  • Robustness: The held-out scenario contained a temporal inconsistency, and a corrected robustness replication reported +73.3 points alongside rather than replacing the original.The correction changed one sentence of the situation text.
  • 4.2 When the constraint is stale, native allocation fails: 77.3%, 74.7%, and 74.7% of superseded-world native episodes followed the stale memory rather than the current record.The counts were 116/150, 112/150, and 112/150 in the primary, replication, and procurement runs.
  • 4.3 Re-allocating the same budget removes most of the error: +74.0 points raised current-record-consistent decisions from 34/150 to 145/150 when one verification slot was reassigned to the critical path.The structural ceiling was 78.7 points, placing the estimate within five points of that ceiling.
  • 4.3 Re-allocating the same budget removes most of the error: The intervention moves one of two existing verification slots rather than adding information, isolating the allocation contribution to stale-memory error.Once the superseding record was in context, decisions followed it in 145/150 intervention episodes and 32/32 native episodes that reached it.
  • 4.4 Fresh wording families: +72.7 points in the fresh-wording replication produced 147/150 current-record-consistent decisions versus 38/150 under native allocation.The contrast was positive in all six models and in every new wording family.

4.5 Original held-out domain

In the procurement held-out domain, reallocating one verification slot to the critical provenance path improved current-record-consistent decisions, but a later audit found a temporal inconsistency in the original scenario. A corrected robustness replication preserved the intervention effect.

  • Original held-out result: +61.3 points current-record-consistent decisions under forced-critical versus native allocation in the procurement world, positive in 6 of six models.The original held-out run gave 130/150 versus 38/150.
  • Scenario inconsistency: The held-out scenario contained conflicting temporal statements about the supply contract’s expiration and onboarding timeline.The frozen text said the contract expired in 3 days, while the source record said onboarding took 6 weeks and an earlier turn said 14 days.
  • Scenario inconsistency: The inconsistency made the record-consistent action partly infeasible on the record’s own numbers and appeared in 10 of 20 non-switching forced-critical rationales.Because the defect affected the intervention arm, its direction was adverse to the reported contrast; the contribution’s size was read post hoc.
  • Robustness replication: +73.3 points in the corrected robustness replication, meeting its pre-specified criterion with 146/150 versus 36/150.The corrected result was reported alongside, rather than averaged with, the original; native stale error was 76.0%.
  • Valid-world control: In the valid world, forced-critical changed Y by only +0.7, +2.0, +0.7, and +0.0 points across the reported conditions.The intervention changed little when the fetched record confirmed the stated constraint.
  • Model heterogeneity: Per-model intervention effects ranged from +16.0 to +96.0 in the primary run, with the magnitude tracking each model’s native under-verification bound.Haiku 4.5 was a zero-effect boundary case because it verified the critical record natively in all 25 corrected-run episodes.

5 Discussion

The paper argues that verification allocation, not merely provenance availability or retrieval relevance, can determine whether stale inherited constraints are corrected. Its randomized fixed-budget intervention identifies a large avoidable error component while leaving the predictor and deployment policy unresolved.

  • Discussion: Under a verification budget, the agent must choose which inherited belief is most costly to leave unchecked, beyond deciding what is relevant or retrieved.The paper separates retrieval’s role in placing information before the agent from verification’s role in checking inherited beliefs.
  • Discussion: Moving one of two slots to the critical path removed most stale-memory error, whereas moving it to a random record did not.The intervention changed nothing when the inspected record agreed with the memory.
  • Discussion: The allocation effect was large, confined to disagreement between memory and record, and bounded by how often the critical path was left uninspected.Except for the inconsistent original held-out run, estimates were within five points of that bound.
  • Discussion: The paper distinguishes a stated-but-wrong constraint from constraint loss under context compaction or prohibition decay over long contexts.Here, the constraint remains present, reads as settled, and is wrong.
  • Architectural implication: Production memory systems may need freshness, supersession, or expected-loss signals separate from semantic relevance, but this paper does not build or evaluate such a scheduler.A deployable policy would need to predict where native allocation under-verifies from observable metadata.
  • Contribution: Relative to prior work, the paper randomizes verification policy and measures downstream error when an inherited constraint is wrong.Its intervention is intended to identify the allocation-attributable share of error, not explain native selection mechanisms.

6 Limitations

The study’s limitations constrain its ecological scope, intervention interpretation, model generality, and replication status. The authors also report a temporal inconsistency in the original held-out scenario and retain that run alongside a corrected robustness replication.

  • The experiments use six-memory stores, two scripted worlds, and one archive semantics, so real stores may involve larger scale and multi-hop discovery.
  • The superseded state is installed by the researchers, and the study does not measure how often stale constraints arise or whether natural consolidation produces them.
  • The forced-critical intervention uses experimenter knowledge of the critical path, identifying avoidable error attributable to allocation but not evaluating a deployable verification policy.
  • The evidence comes from six models from two providers, with effect magnitude governed by each model’s native verification rate; direction generalizes more consistently than magnitude.
  • The design is task- and schema-specific, includes residual forced-versus-native asymmetries, and does not identify mediation or the mechanism of native under-verification.
  • The original held-out scenario contained a temporal inconsistency; the corrected replication was specified and deposited before execution but remains post hoc relative to the original run.
  • The study does not evaluate a scheduler, natural consolidation chains, or independent-team replication, although episode files, manifests, scripts, and AI-assistance disclosures are provided.

A Full arm tables, per-model and per-family results

The result tables show recovery across directions and robustness checks, while the original and corrected held-out runs are explicitly presented as separate analyses. The intervention’s benefit is broad across models but has no effect in a cell where memory and current record already agree.

  • Per-model results: No single model carries the result: leave-one-model-out estimates remain at or above +69.6 points in primary, +68.0 in replication, +56.0 in original held-out, and +68.0 in corrected held-out runs.
  • Control cell: 129/150 native and 139/150 forced-critical decisions were current-record-consistent in the primary no-constraint superseded cell, with little change from fetching.
  • Full arm results: 145/150 primary and 147/150 replication decisions followed a record withdrawing a constraint, while the installing direction reached 150/150 in both runs.
  • Held-out results: The original held-out run reports +61.3 points, whereas the corrected robustness replication reports +73.3 points and supplements rather than replaces the original.

B Prospective specification and external registration record

The study separates prospective evidence for the original runs from a separately frozen and externally deposited corrected robustness package. The reported per-model and per-family tables frame direction as generalizing while magnitude depends on native verification rates and wording families.

  • Prospective record: The primary, replication, and original held-out packages were frozen, hashed, committed, timestamped before confirmatory episodes, and run only after the registration record was present.
  • Corrected package: The corrected held-out package was separately frozen, hashed, externally deposited to OSF, and verified byte-for-byte before execution.
  • Per-model results: The per-model table defines the same-budget effect as forced-critical minus native on Y and compares it with each model’s structural bound.
  • Per-family results: The per-wording-family table reports forced-critical versus native Y across six primary and six fresh replication wording families.
  • Verification criteria: All three corrected-run criteria were met, including a positive intervention comparison against the original run’s 86.7% consistent-decision rate and blinding checks.
  • Reproducibility: An independent implementation reproduces every count and interval for all four runs from the released episode files and frozen materials.

C The held-out inconsistency and the corrected robustness protocol

The original held-out procurement scenario contained a temporal inconsistency that could depress the intervention arm. A one-sentence correction was specified, audited, and executed as a separate robustness replication, whose effect remained positive and larger.

  • Original inconsistency: The frozen procurement scenario stated a three-day contract deadline, while the source and turn-1 context supplied earlier timing information.
  • Discovery: Ten of twenty forced-critical non-switching rationales accepted the superseding record but cited the onboarding, deadline, and production-buffer context.
  • Bias direction: The inconsistency could only lower the intervention arm’s outcome in the superseded world, so its direction was adverse to the reported effect; the 10–13-point attribution was post hoc.
  • Correction: The corrected text changed the deadline to 11 days and added a month-to-month bridge during transition, with all other exported prompts, records, schema, and runner sources unchanged.
  • Robustness result: +73.3 points [ +68.7, +77.3 ] were obtained in the corrected run, compared with +61.3 [ +54.0, +68.0 ] originally; the corrected interpretation was fixed before execution.

D The forced-noncritical control and its design limitation

The forced-noncritical control compares critical-path reassignment with a random-record alternative, but its contrast partly measures the cost of overriding native choices rather than the critical record’s value.

  • The forced-noncritical control: +85.3, +87.3, +74.7 and +83.3 points were the forced-critical minus forced-noncritical contrasts across the primary, replication, original held-out and corrected held-out runs.These contrasts include roughly 11–14 points attributable to overriding a native choice.
  • The forced-noncritical control: The noncritical arm discarded the target when the agent named it second, occurring in 19, 18, 19 and 21 episodes across the four runs.Every such episode was stale-consistent.
  • Design limitation: The authors therefore withdraw narrative weight from this contrast because it measures the cost of overriding a native choice, not solely the value of the critical record.The pre-specified rule for the contrast was met, but the authors do not use it as a headline result.
  • Design limitation: A repaired comparator preserving the agent’s own target pick while replacing exactly one slot was specified but not run.The authors state that the headline contrast does not require this comparator.

E Reproducibility and audit details

The study reports balanced, scripted runs with extensive integrity checks, predetermined analysis procedures, and six named models accessed through structured-output APIs.

  • Runs and units: 5,400 confirmatory episodes comprised 1,800 primary, 1,800 replication, 900 original held-out and 900 corrected held-out episodes.The runs produced 10,800 kept model calls; no episode or model was excluded.
  • Runs and units: The unit of analysis was one episode containing one turn-1 allocation and one turn-2 decision for one model.A 48-episode pilot was excluded by version string.
  • Integrity checks: Integrity checks passed on every run, including exact cell counts, unique keys and seeds, byte-identical within-block prompts, and valid responses.Checks also verified that superseding records and status lines were absent from turn-1 prompts.
  • Analysis: Risk differences used equal-weight per-model arm means, with percentile bootstrap intervals over episodes within each model and arm.Each frozen analysis script was executed once on its completed run; Cochran–Mantel–Haenszel tests were corroborative, not headline results.
  • Models and interfaces: Six models were evaluated: Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna.Anthropic models used the Messages API; OpenAI models used the Responses API with medium reasoning effort and strict JSON schema.
Loading 2608.25553v1…