PaperlayerView exact revision on arXiv

Source-linked AI summary

SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

arXiv:2609.33436v1cs.LG

TL;DR

Bounded recurrent models must preserve selected updates through subsequent inputs, a capability not established by immediate retrieval or performance on learned sequence distributions. SchemaMem combines chunk-local attention with schema-indexed recurrent state, and controlled comparisons find stronger long-delay written-value retention than the tested baselines, alongside optimization-cost and scope limitations.

  • Problem

    Whether selected updates survive long streams of intervening inputs remains distinct from performance on learned sequences, and bounded-state recurrence must encode information that full-history attention can revisit directly.

    Source: §1 · p. 1

  • Method

    SchemaMem combines chunk-local attention with a learned schema reference and episode-specific phase state, using static-schema writes and commits to carry updates across chunks.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    At delay 1,536, SchemaMem has higher mean retention accuracy than both baselines across all nine settings, while updated-value recovery favors it in nine settings against Mamba-3 and seven against Gated DeltaNet.

    Source: §5 · p. 5

  • Takeaways & Limitations

    The experiments identify a long-delay retention regime at comparable parameter and persistent-state sizes, but support a design trade-off rather than interference-free storage.

    Source: §6 · p. 7

    The comparison is limited by small models, three seeds per setting, differing training procedures and optimization costs, and a narrow family of synthetic episodes.

    Source: §6 · p. 7

Abstract

from arXiv · show

Attention provides direct access to past representations, but retaining an ever-growing history is costly. Recurrent models bound persistent state, yet must preserve selected information while processing subsequent inputs. We introduce SchemaMem, an attention-based recurrent memory architecture combining chunk-local attention with a persistent, schema-indexed phase state. Learned schema embeddings provide a shared representational reference for reading and writing. Reads use the current state, whereas writes use the layer input and static schema embeddings, excluding direct feedback from that layer's own state. Chunk-boundary commits aggregate bounded phase increments through forward computation. The same parameters also support full-history attention training before and during recurrent training. We studied selective updates, preservation, and delayed retrieval in a controlled address--value task, comparing three-layer models with approximately matched parameter counts and persistent-state dimensions. Across nine address/value settings and three training seeds, SchemaMem has higher mean written-value retention at four times the maximum training delay than both baselines, which are trained toward a higher in-range accuracy target. Updated-value recovery favors SchemaMem in all nine settings against Mamba-3 and seven against Gated DeltaNet. Defaults consistently favor Gated DeltaNet over SchemaMem at that delay, and SchemaMem requires substantially more optimization steps. These results identify a promising retention--optimization trade-off in schema-indexed recurrence.

1 INTRODUCTION

Bounded-state recurrence must preserve selected updates through later inputs, unlike full-history attention, which can revisit original observations. SchemaMem combines chunk-local attention with a learned schema bank and episode-specific phase state; its long-delay retention advantage comes with substantial training-cost disadvantages.

1 INTRODUCTION

  • Longer intervening streams matter because good retrieval immediately after a write does not establish that a bounded recurrent state can preserve its consequences.Full-history attention can consult original observations, whereas bounded-state recurrence must encode their consequences in a representation that survives later computation.Source: §1 · p. 1
  • Sequence-distribution performance does not determine whether selected updates survive longer streams, motivating a controlled task with exact answers that isolates address–value preservation and revision.The task avoids conflating those capabilities with language comprehension or generation.Source: §1 · p. 1
  • SchemaMem combines chunk-local attention with a learned schema bank and episode-specific phase state, using the bank as a shared reference for reading and writing.The static bank contributes a learned representation even when the episode state is zero; inference-time state changes the bank without changing model parameters.Source: §1 · pp. 1–2
  • The design separates recent processing from cross-chunk communication: local attention handles recent representations, while persistent state carries information across chunks.The writer excludes same-layer state, while lower-layer memory can still influence higher-layer writes through hidden representations.Source: §1 · p. 2
  • At substantially longer delays, SchemaMem retains more written values than baselines that can be more accurate within the training range, though the reversal is not uniform at higher loads.The reversal is clearest in the 8- and 16-address settings and comes with a large training-cost disadvantage.Source: §1 · p. 2
  • The contributions include a controlled evaluation that separates unchanged written values, new values, and never-written defaults, and tests whether in-range accuracy predicts delayed retention.The paper also identifies optimization and capacity limitations of the proposed alternative.Source: §1 · p. 2

2 RELATED WORK

Prior work includes external differentiable memory, learned persistent vectors, segment recurrence, compressed histories, and block-recurrent attention, as well as recurrent state-space and fast-weight models. SchemaMem’s stated distinction is its combination of a learned schema reference, episode-specific phase residuals, static-schema writing, and additive boundary commits.

Attention with persistent and recurrent memory

  • SchemaMem’s distinction from prior persistent and recurrent attention approaches is the combination of a learned schema reference, episode-specific phase residuals, a static-schema writer, and additive boundary commits.Related approaches include differentiable external memory [Graves et al., 2014], learned persistent vectors [Sukhbaatar et al., 2019], segment recurrence and compression [Rae et al., 2019], recurrent memory tokens, and block-recurrent attention [Hutchins et al., 2022].Source: §2 · p. 2

State-space and fast-weight sequence models

Memory persistence as a separate evaluation axis

  • MemMamba motivates measuring persistence after intervening computation rather than relying on immediate retrieval alone [Wang et al., 2025].This study separates kinds of state changes and reports exact-value as well as bit accuracy.Source: §2 · p. 2

3 SCHEMA-INDEXED RECURRENT MEMORY

Each layer combines a learned schema bank with a chunk-specific phase state: reads use their combination, while writes query the static bank from the pre-read input. Bounded phase increments are aggregated at chunk boundaries, making updates available to the next chunk without direct same-layer state feedback to the writer.

3 SCHEMA-INDEXED RECURRENT MEMORY

  • Each layer has an independent learned schema bank and an episode-specific phase state of the same shape; the state starts at zero and remains fixed within each chunk.A schema slot is not assigned to a database address, and the state is not a model parameter.Source: §3 · p. 2
  • Reads use the combined phase E + Sc, while writes use pre-read input and static phase E, excluding direct same-layer state feedback.The figure describes local attention as unable to access earlier chunks’ keys or values; commits make updates readable in the next chunk.Source: Figure 1 · p. 3, §3 · p. 4
  • The read branch uses dense attention over schema slots, and even at Sc = 0 it returns an input-dependent mixture of learned static schema values.Read and write branches share key/value projections but use distinct query projections.Source: §3 · p. 3
  • Writing distributes bounded per-position phase increments across slots, sums them over the chunk, and commits the accumulated angles modulo 2π for the next chunk.The update is summed rather than averaged over chunk length; the bound applies to individual increments, not the total chunk update.Source: §3 · p. 3
  • Within a chunk, ordinary causal attention processes X + R without previous chunks’ keys or values, and the layer output also uses the input residual and a projection of write features.The write-feature residual supplies a training signal before a later chunk reads the committed state.Source: §3 · p. 3
  • Persistent state scales as LMd elements, with fixed slot count and chunk length yielding linear work in sequence length but sequential chunk dependencies.The table caption notes that persistent-element counts exclude temporary buffers and local attention storage.Source: §3 · p. 4, Table 1 · p. 4

4 CONTROLLED STATE-MAINTENANCE EXPERIMENTS

The controlled task tests address–value maintenance through intervening queries, distinguishing retained, overwritten, and default values. Models are compared under approximately matched parameter and persistent-state counts, with different training procedures and termination targets; evaluation extends to four times the maximum training delay.

4 CONTROLLED STATE-MAINTENANCE EXPERIMENTS

  • The task uses 2^a addresses and 2^v values, with learned defaults, write transactions, and NULL queries; exact-value accuracy requires every value bit to be correct.Only query positions receive classification supervision, and answers are not fed back as inputs.Source: §4 · p. 4
  • Maintenance scenarios distinguish retained written values, updated values, and untouched defaults after intervening NULL queries, which test persistence through computation rather than pure time decay.The scenarios include disjoint writes and overwrites, and final reads cover the entire address space.Source: §4 · p. 4
  • All models use three layers and width 32, with approximately matched parameters and persistent elements; SchemaMem commits every 96 inputs while baselines scan without state resets.The accounting does not match total working memory, bytes, or FLOPs.Source: §4 · p. 4
  • Training increases queried subsets and maximum delays through 96, 192, and 384; SchemaMem alternates full-history and recurrent batches, while baselines train recurrently throughout.Training stops at 95% validation accuracy for SchemaMem or 99% for baselines, so the baseline control targets higher in-range mastery rather than equal compute.Source: §4 · p. 5
  • Evaluation covers delays through 1,536 across nine address/value settings and three seeds, using recurrent terminal checkpoints and 512 episodes per scenario.The longest delay is four times the maximum training delay; the evaluation generator seed is reused from validation, making these controlled diagnostics rather than a separately held-out test set.Source: §4 · p. 5

5 RESULTS

Across nine settings, SchemaMem retains more written values than both baselines at delay 1,536, despite weaker in-range results in some comparisons and substantially higher optimization-step counts. Its advantage is conditional: defaults and some higher-load updated-value results favor Gated DeltaNet.

5 RESULTS

  • At delay 1,536 in A3V3, updated-value accuracy is 74.64% for SchemaMem, 26.89% for Mamba-3, and 27.15% for Gated DeltaNet.At delay 384, the corresponding accuracies are 95.35%, 99.58%, and 99.74%.Source: §5 · p. 5, Figure 2 · p. 6
  • At delay 1,536, SchemaMem has higher mean retention accuracy than both baselines across all nine settings, with and without disjoint writes.Updated-value recovery favors SchemaMem in nine settings against Mamba-3 and seven against Gated DeltaNet; comparisons are three-seed means, not independent significance tests.Source: §5 · p. 5
  • Defaults favor Gated DeltaNet over SchemaMem in every grid cell; in A3V4, Gated DeltaNet also leads at delay 768, 95.18% to 85.90%.In A3V5 at delay 1,536, SchemaMem obtains 64.78% on defaults versus 96.57% for Gated DeltaNet.Source: §5 · p. 6
  • At delay 1,536, Gated DeltaNet leads SchemaMem on updated values in A5V3, 70.43% to 64.26%, and A5V4, 75.37% to 64.49%.A5V5 is close: SchemaMem scores 67.40% versus 65.80% for Gated DeltaNet, with substantial seed variation.Source: §5 · p. 7
  • Zeroing SchemaMem’s phase state after each commit reduces retained- and updated-value accuracy below 1% for all 27 checkpoints at delays 384 and 1,536.Default predictions largely survive and improve at delay 1,536 for every checkpoint.Source: §5 · p. 7
  • In A3V3, SchemaMem takes 88,667 mean optimizer steps, versus 3,917 for Mamba-3 and 3,750 for Gated DeltaNet.Counts include attention-assisted batches and are not equal-FLOP budgets.Source: §5 · p. 7

6 DISCUSSION AND LIMITATIONS

The experiments identify a long-delay retention regime for SchemaMem, while the discussion emphasizes diagnostic value and limits on what the results establish. The proposed design remains costly to optimize, and transfer to stateful-agent settings is prospective rather than demonstrated.

6 DISCUSSION AND LIMITATIONS

  • SchemaMem loses fewer written values after prolonged intervening computation than the tested small Mamba-3 and Gated DeltaNet models, but this is a trade-off rather than evidence of interference-free storage.The result is scoped to comparable parameter and persistent-state sizes.Source: §6 · p. 7
  • Separating retained, overwritten, and default values exposes distinct failure profiles that aggregate accuracy can conceal.The controlled task increases intervening queries without adding language comprehension, answer generation, or ambiguous ground truth as variables.Source: §6 · p. 7
  • The learned schema is a design motivation, not an established cause of the retention advantage; isolating its role would require ablations and matched training schedules.The S = 0 control confirms dynamic state is used for written-value retrieval, but does not identify which design component explains the advantage.Source: §6 · p. 7
  • SchemaMem’s stronger retention does not erase its slow, seed-sensitive optimization, and whether matched schedules or alternative curricula narrow the gap remains open.The baselines reach higher in-range targets, yet retain a long-delay deficit in the identified regime.Source: §6 · p. 7
  • The comparison uses small models, three seeds per setting, differing training procedures, and a narrow synthetic episode family; arbitrary interleavings, deletions, random defaults, larger databases, and continuous streams remain untested.Delayed NULL queries are a specific distractor distribution, and structured write/read boundaries are supplied by the experiment.Source: §6 · pp. 7–8
  • The results motivate testing retention in agent interactions, but the study does not measure effects on end-to-end agent performance.Using external records alongside learned state is presented as a prospective application, not a demonstrated capability.Source: §6 · p. 8

7 CONCLUSION

SchemaMem combines chunk-local attention with learned schema and episode-specific phase state, and controlled experiments reveal a promising long-delay retention regime with optimization and capacity weaknesses. The conclusion frames these findings as a basis for further controlled work, not a universal memory claim.

7 CONCLUSION

  • SchemaMem combines chunk-local attention with a learned schema reference and episode-specific phase state, with encouraging long-delay retention alongside weaknesses in optimization cost, defaults, and some higher-load conditions.The study presents a controlled basis for investigating recurrent state maintenance and calls for isolating the architectural and training choices behind the trade-off.Source: §7 · p. 8
  • The reported numerical results come from executed training and evaluation runs, with checks including code inspection, executable tests, and table crosschecks against saved evaluation records.The authors take responsibility for the final text, claims, code, and results.Source: AI USE STATEMENT · p. 9
  • The experiments use programmatically generated address–value sequences and test narrow state maintenance, not autonomous-agent safety or reliability.Applying persistent memory to human-facing systems would add privacy, consent, and deletion requirements not evaluated here.Source: ETHICS STATEMENT · p. 9
  • The supplementary workflow provides implementation, task generators, configurations, scripts, plotting functionality, and setup instructions, but not trained checkpoints.The workflow trains models from scratch and evaluates the resulting checkpoints.Source: REPRODUCIBILITY STATEMENT · p. 9

A.1 COMPLETE LONG-DELAY COMPARISONS

The complete long-delay comparison covers four prediction groups across nine settings, while the reported results show that outcomes vary by prediction group and setting. SchemaMem's retention advantage does not imply it wins every seed or group.

A.1 COMPLETE LONG-DELAY COMPARISONS

  • The comparison holds hidden width, slot count, depth, and baseline state sizes fixed as address and value widths vary; exact-value accuracy requires every value bit to be correct.Increasing value width makes exact-value accuracy more demanding even at fixed bit accuracy.Source: §A.1 · p. 11
  • Table 3 reports exact-value accuracy at delay 1,536 for Retained-R, Retained-I, Updated, and Default across all 81 endpoints.The four prediction groups cover retention-only, retained values after disjoint writes, and updated and default values from overwrite scenarios.Source: Table 3 · p. 11
  • Gated DeltaNet outperforms SchemaMem on updated values in A5V3 and A5V4 and on defaults in all nine settings, while A5V5 updated-value means are 67.40 ± 4.31% versus 65.80 ± 26.62%.For A4V3 updated-value accuracy across seeds 42/43/44, SchemaMem is 57.03/84.23/91.85% and Gated DeltaNet is 79.10/29.20/0.93%.Source: §A.1 · p. 11

A.2 TRAINING COST

The appendix reports optimizer-step costs, full-grid delay profiles, and seed-level endpoints. Long-delay advantages are qualified by scenario, seed, and training-budget differences.

A.2 TRAINING COST

  • SchemaMem requires substantially more training, and its optimizer-step totals include full-history batches without matched-FLOP or equal-example-length budgets.The comparison does not support a training-efficiency advantage.Source: §A.2 · p. 12, Table 4 · p. 12

A.3 COMPLETE DELAY PROFILES AND WITHIN-SEED DEGRADATION

  • At the longest delay, retained written-value means favor SchemaMem in all nine cells, although individual seeds and shorter delays do not uniformly favor it.Defaults favor Gated DeltaNet in every cell at delay 1,536; written-value retention does not establish preservation of every useful computation.Source: Figure 4 · p. 14, Figure 6 · p. 16
  • Figure 7 summarizes within-seed exact-accuracy changes from delay 384 to 1,536, but these changes are neither causal estimates of forgetting nor matched-accuracy comparisons.A smaller drop need not imply higher absolute accuracy.Source: §A.3 · p. 13, Figure 7 · p. 17

B.1 SEED-LEVEL VARIATION

  • The seed-level display retains large seed effects, and all 81 runs reach their configured final targets with no seed removed from aggregates.Table 5’s pass status refers to the configured final target, not long-delay accuracy.Source: §B.1 · p. 18, Table 5 · p. 18, Figure 8 · p. 20

B.2 SCENARIO-LEVEL RESULTS

Table 6 reports final-query exact-value accuracy by maintenance scenario at delay 1,536, averaged across three training seeds with sample standard deviation.

B.2 SCENARIO-LEVEL RESULTS

  • Table 6 covers retained, updated, and default-value groups across maintenance scenarios at delay 1,536; intermediate and delay-filler queries are excluded.Values are exact-value accuracy, reported as mean ± sample SD across three training seeds.Source: §B.2 · pp. 21–22, Table 6 · p. 21

C DYNAMIC-STATE ABLATION

The dynamic-state ablation clears every layer’s state after each commit while preserving the other listed model components. Written-value accuracy collapses, while default predictions improve at the longest delay.

C DYNAMIC-STATE ABLATION

  • With S set to zero after every commit, retained- and updated-value accuracy falls below 1% for every checkpoint at both delays.Normal and ablated evaluations use the same checkpoints and episodes, without retraining.Source: C DYNAMIC-STATE ABLATION · p. 23, Table 7 · p. 23, Table 8 · p. 24
  • At delay 1,536, default accuracy rises from 64.15% normally to 99.51% with S = 0, improving in every checkpoint.At delay 384, overwrite-scenario default accuracy averages 99.83% normally and 99.52% with S = 0.Source: C DYNAMIC-STATE ABLATION · p. 23
  • Because written values differ from defaults, reverting to the learned default mapping does not recover episode-specific answers.The intervention supports dynamic-state use for conveying written values but does not isolate learned schema embeddings from other memory parameterizations.Source: C DYNAMIC-STATE ABLATION · p. 23

Related papers

Loading 2609.33436v1…