PaperlayerView exact revision on arXiv

Source-linked AI summary

A Persistent State for Auditable Mixture-of-Experts Routing

Abdurrahman Javat, Allan Kazakov

arXiv:2609.34634v1cs.AI

TL;DR

Conventional MoE routers expose no routing-specific record of how cross-layer influences accumulate. SA-MoE gives routers a persistent state whose realized writes decompose its contribution to later routing margins; across two model families, persistence improves long-horizon routing accessibility over a matched latest-write-only control. The ledger audits only this state pathway, not all influences on routing.

  • Problem

    Conventional MoEs expose no routing-specific record of how cross-layer routing influences accumulate, although later routing may depend on influences across depth.

    Source: §1 · p. 1

  • Method

    SA-MoE gives routers a low-dimensional persistent state, unavailable to experts, whose realized writes provide an exact decomposition of the state-mediated contribution to later routing margins.

    Source: §1 · p. 1

  • Results

    Persistence increases long-horizon future-routing accessibility over a matched latest-write-only control by 19.4 and 12.2 percentage points in SmolLM2- and Gemma-based models, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SA-MoE makes one cross-layer routing pathway explicitly auditable, and full-forward suppression shows ledger-selected writes can affect subsequent routing and model outputs.

    Source: §7 · p. 9

    The ledger audits only the persistent routing-state pathway and is not a complete causal explanation of expert selection; experiments on two model families do not establish behavior at larger MoE scales.

    Source: §6 · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models repeatedly route tokens to sparse subsets of experts, but conventional routers expose no routing-specific record of how cross-layer influences accumulate. We introduce Scratchpad-Augmented Mixture-of-Experts (SA-MoE), which gives each router access to a low-dimensional persistent state that is not provided to the experts. Learned layerwise writes update this state, and their realized post-update changes exactly decompose the state-mediated contribution to any later routing margin, forming a routing ledger. Across sparsely upcycled SmolLM2- and Gemma-based models and three independent training seeds per architecture, this pathway adds less than 1% analytical forward compute and is strongly used by trained routers: local removal of its router contribution changes the selected Top-2 expert set in 87.6% and 69.9% of decisions, respectively. Relative to a matched latest-write-only control, persistent accumulation increases long-horizon future-routing accessibility by 19.4 and 12.2 percentage points, with positive effects in every seed. More than 90% of absolute ledger contribution comes from non-recent writes in both families, and full-forward suppression of ledger-selected writes changes later routing and output distributions. The ledger is an exact provenance object for the persistent-state pathway, not a complete causal explanation of routing. Sensitivity-aware scores better predict full-forward intervention effects, and post-hoc methods recover related cross-layer attribution without architectural modification. SA-MoE instead makes one routing-specific computational history explicit and directly inspectable within the model's natural forward computation.

1 INTRODUCTION

Conventional MoEs lack a routing-specific record of how cross-layer influences accumulate. SA-MoE makes one such pathway explicit through persistent state and write-level provenance, and reports that trained routers use it and that persistence improves long-horizon routing accessibility.

1 INTRODUCTION

  • Conventional routers expose no routing-specific record of cross-layer influences accumulating across depth, although such influences can be reconstructed post hoc.Prior architectures have introduced explicit memory between routers, but do not make write-level routing provenance a first-class forward-computation object.Source: §1 · p. 1
  • SA-MoE gives each router a low-dimensional persistent state unavailable to experts, and realized state changes exactly decompose its contribution to later routing margins.Learned write events form the native provenance object and support source-level intervention.Source: §1 · p. 1
  • Persistence improves long-horizon future-routing accessibility over a matched latest-write-only control by 19.4 and 12.2 percentage points in SmolLM2- and Gemma-based models, respectively.The contrast is positive across all three seeds in both families, and more than 90% of absolute ledger contribution comes from non-recent writes.Source: §1 · p. 2
  • SA-MoE makes one cross-layer routing pathway explicitly inspectable through persistent state, write-level provenance, retrospective queries, and interventions on recorded source events.The paper presents this as complementary to recurrent routing memory, hidden-stream probing, and post-hoc attribution rather than as a replacement.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior work carries routing information across layers, interprets MoE structure, or reconstructs routing influences after training. SA-MoE differs by making learned writes a native routing-provenance object and by distinguishing attribution from intervention sensitivity.

2.1 SPARSE MOE ROUTING AND CROSS-LAYER MEMORY

  • RMoE carries routing-specific information between successive MoE layers, while exact telescoping of a recorded recurrent state is not unique to SA-MoE.SA-MoE instead exposes learned writes as an audit interface that can be queried or suppressed at the source; Memory-Aware Routing targets stable expert specialization.Source: §2.1 · p. 2

2.2 ARCHITECTURE-LEVEL MOE INTERPRETABILITY

  • MoE-X and MONET target sparse expert computation or interpretable expert structure, whereas SA-MoE audits routing-pathway provenance without requiring semantic meanings for experts or state coordinates.Source: §2.2 · p. 2

2.3 POST-HOC ATTRIBUTION AND CAUSAL VALIDATION

  • Post-hoc methods reconstruct cross-layer routing influences in ordinary MoEs, while SA-MoE places routing-specific state and write events inside the natural forward computation.The paper evaluates Li et al. (2026) on matched Ordinary MoEs rather than claiming attribution requires a dedicated routing state.Source: §2.3 · p. 2
  • Full-forward source suppression tests whether selected components affect behavior, while gradient-based rankings provide a complementary sensitivity-aware comparison.The experiments distinguish exact forward provenance, post-hoc attribution, and intervention sensitivity rather than treating them as the same objective.Source: §2.3 · p. 3

3 SCRATCHPAD-AUGMENTED MIXTURE-OF-EXPERTS

SA-MoE routes using both the ordinary token representation and a routing-specific state that experts do not receive. Its accumulated realized writes exactly decompose the persistent-state contribution to later router margins, while ordinary hidden-stream influences remain outside that ledger.

3 SCRATCHPAD-AUGMENTED MIXTURE-OF-EXPERTS

  • Each router reads the ordinary representation and routing state, but experts read only the ordinary representation.A learned write head updates the routing state after each MoE layer.Source: §3 · p. 3, Figure 1 · p. 4
  • The routing ledger exactly decomposes the persistent-state contribution to a later router margin into contributions from earlier realized writes.The decomposition does not include influences mediated through the ordinary hidden stream.Source: §3 · p. 3
  • Latest-write-only preserves state width and write-head capacity but replaces rather than accumulates state, so its native audit object is the currently retained latest write.Ordinary MoE removes the routing-state pathway entirely.Source: §3 · p. 3

4 EXPERIMENTAL SETUP

The experiments compare matched sparse architectures based on two sparsely upcycled model families, with capability, state-use, persistence, accessibility, and intervention evaluations. Training-seed replication is kept distinct from sequence-level uncertainty for within-checkpoint interventions.

4 EXPERIMENTAL SETUP

  • The study trains Ordinary MoE, latest-write-only, and Persistent SA-MoE variants for SmolLM2-135M and Gemma-270M, using three independent seeds per architecture and matched continued-pretraining budgets.The sparse backbones use 16 routed experts, one shared expert, Top-2 routing, and approximately 30B packed input tokens; state-augmented variants use a 128-dimensional routing state.Source: §4 · p. 4
  • Capability evaluation measures validation perplexity and the unweighted macro average of four zero-shot tasks, using non-inferiority thresholds of at most 1% relative perplexity increase and 1 percentage point macro-score decrease versus Ordinary MoE.Source: §4 · p. 4
  • State-use tests locally remove the routing-state contribution while holding the ordinary hidden representation fixed, then measure changes in selected Top-2 sets; persistence is compared with the matched latest-write-only control.Source: §4 · p. 4
  • Future-routing accessibility probes predict a later router’s Top-1 expert from the native state, a full-hidden linear probe, or a supervised 128-dimensional hidden subspace.Probe-training budgets include 10^4, 10^5, and 10^6 examples.Source: §4 · pp. 4–5
  • The intervention study audits 96 target routing decisions per family and compares ledger, recency, write-norm, gradient×write, and post-hoc attribution rankings before full-forward source suppression.It measures audited-margin change, later Top-2 routing divergence, and output-distribution KL divergence; raw intervention magnitudes are not compared across architectures.Source: §4 · p. 5
  • Architecture-level uncertainty uses paired seed-level contrasts and three-seed 95% Student-t intervals, while within-checkpoint intervention uncertainty uses paired sequence-level bootstrap resamples.Bootstrap samples are not treated as additional training replicates.Source: §4 · p. 5

5 RESULTS

The results show that trained routers use SA-MoE’s state pathway and that persistent accumulation improves long-horizon routing accessibility. The ledger records state-pathway provenance, but does not establish an optimal causal ranking of writes.

5.1 MODEL QUALITY AND ARCHITECTURAL COST

  • The pathway adds less than 1% active parameters and analytical forward FLOPs per token; SmolLM2 met its capability criterion, while Gemma’s downstream capability result remained inconclusive.SmolLM2’s four-task macro score changed by −0.05 pp; both Gemma downstream macro intervals crossed the pre-specified −1 pp boundary.Source: §5.1 · pp. 5–6

5.3 PERSISTENCE IMPROVES LONG-HORIZON ROUTING ACCESSIBILITY

  • Persistent raises long-horizon routing accessibility over latest-write-only by 19.4 pp on SmolLM2 and 12.2 pp on Gemma, positive in all three seeds.The reported confidence intervals are [13.2, 25.6] pp for SmolLM2 and [5.4, 18.9] pp for Gemma.Source: Figure 2 · p. 6, §5.3 · p. 7

5.2 TRAINED ROUTERS USE THE PERSISTENT STATE

  • At full local state suppression, selected Top-2 expert sets change for 87.6% of SmolLM2 and 69.9% of Gemma routing decisions.This intervention holds the hidden representation and state fixed and does not rerun later computation, measuring direct dependence of the current routing decision.Source: §5.2 · p. 6

5.4 COMPACT ACCESS TO FUTURE ROUTING

  • At 10^6 examples, Persistent native state exceeds latest-write-only by 22.5 pp on SmolLM2 and 10.4 pp on Gemma.Figure 3 compares future-router Top-1 prediction accuracy across probe-training budgets for both native states and a supervised hidden-stream subspace.Source: Figure 3 · p. 7, §5.4 · p. 7

5.3 PERSISTENCE IMPROVES LONG-HORIZON ROUTING ACCESSIBILITY

  • Non-recent writes account for 91.7% of absolute ledger contribution on SmolLM2 and 90.3% on Gemma, with effective source supports of 9.49 and 5.82 writes.These measure absolute forward contribution, not signed net support or downstream causal importance.Source: §5.3 · p. 7

5.4 COMPACT ACCESS TO FUTURE ROUTING

  • At the largest budget, the supervised hidden subspace reaches 83.4% and 84.1% accuracy, versus 81.2% and 77.4% for Persistent native state on SmolLM2 and Gemma.Future-routing information is therefore not uniquely available in the native state; its distinction is being present as a compact routing-specific representation in the forward state.Source: §5.4 · pp. 7–8

5.5 ATTRIBUTION QUALITY AND DOWNSTREAM CONSEQUENCES

  • Persistent ledger-selected-write suppression changes audited margins by 0.276 and 0.426, downstream Top-2 divergence by 0.080 and 0.195, and output KL by 0.0016 and 0.0162 on SmolLM2 and Gemma.Latest-write-only interventions also produce downstream effects, and Persistent-minus-latest-write-only contrasts are not uniformly resolved; provenance is not an optimal causal ranking.Source: Figure 4 · p. 8, §5.5 · pp. 8–9
  • Ledger scores correlate more strongly with full-forward intervention effects than largest-write-norm scores, while gradient×write correlations are higher still.Within-target Spearman correlations are 0.369 versus 0.187 on SmolLM2 and 0.823 versus 0.341 on Gemma; on matched Ordinary MoEs, Li et al. (2026) post-hoc attribution reaches 0.570 and 0.620, but the audit objects differ.Source: §5.5 · p. 8

6 DISCUSSION AND LIMITATIONS

SA-MoE makes realized writes an explicit provenance interface for the persistent-state contribution to later routing margins. This ledger is not a complete causal account of routing, and the reported experiments leave scale and intervention uncertainty unresolved.

6 DISCUSSION AND LIMITATIONS

  • The ledger exactly decomposes the persistent-state contribution to a realized routing margin, but does not cover hidden-stream influences or guarantee the strongest intervention ranking.Gradient×write incorporates downstream sensitivity; post-hoc attribution can recover related influences without architectural modification.Source: §6 · p. 9
  • The latest-write-only comparison confounds accumulation with state geometry because replacement is bounded to [−1, 1] while Persistent can accumulate toward the [−5, 5] clamp.The comparison therefore cannot separate accumulation from differences in state magnitude and saturation.Source: §6 · p. 9
  • Experiments cover two sparsely upcycled model families and three seeds per architecture, leaving some intervention contrasts uncertain and behavior at larger MoE scales unestablished.The ledger also does not assign semantic meaning to state coordinates or imply improved generation quality or safety.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents SA-MoE as an explicitly auditable routing pathway, while the appendices specify model construction, training, and implementation choices. Together, these details delimit the reported architecture and experimental setup.

7 CONCLUSION

  • Across two sparsely upcycled model families and three training seeds, routers use the persistent pathway, accumulation improves long-horizon routing accessibility over a matched latest-write-only control, and ledger contributions are distributed across earlier writes.Full-forward suppression shows ledger-selected writes can affect subsequent routing and model outputs.Source: §7 · p. 9

A MODEL, TRAINING, AND REPRODUCIBILITY

A.1 DENSE SOURCE MODELS AND TOKENIZER PROVENANCE

  • The experiments use sparsely upcycled SmolLM2-135M and Gemma-3-270m checkpoints, with byte-compatible reproducible model and tokenizer revisions documented.Each tokenizer comes from the same repository revision as its model.Source: §A.1 · pp. 12–13

A.2 MOE ARCHITECTURE

  • Each MoE block has 16 routed experts and one always-active shared expert, and each token executes the shared expert plus its Top-2 routed experts.Ordinary MoE routes from xℓ alone; state-augmented variants route from the concatenation of xℓ and sℓ.Source: §A.2 · p. 13

A.3 SPARSE UPCYCLING AND INITIALIZATION

  • Routed experts begin as copies of the pretrained dense MLP and are partially reinitialized independently, while the shared expert remains an unchanged copy.The principal experiments use three independent training seeds per family and architecture variant.Source: §A.3 · pp. 13–14

A.4 PERSISTENT AND LATEST-WRITE-ONLY STATE IMPLEMENTATIONS

  • Persistent state begins at zero and is threaded through MoE layers; latest-write-only replacement has a native state range of [−1, 1], unlike Persistent’s wider clamp.The write head reads the shared-expert representation; the latest-write-only model has no separately retained accumulator.Source: §A.4 · p. 14

A.5 CONTINUED-PRETRAINING RECIPE

  • Training samples come from FineWeb-Edu, The Stack Dedup Python, and FineMath-4+ with sampling probabilities 0.50, 0.25, and 0.25, respectively.Runs use 30,000 optimizer steps; Gemma latest-write-only uses a different microbatch and accumulation shape while preserving the global batch and nominal token budget.Source: §A.5 · pp. 14–15

A.6 PARAMETER COUNTS AND ANALYTICAL COMPUTE

This appendix reports analytical model-size and active-compute accounting, then describes runtime diagnostics and reproducibility and evaluation protocols. It also specifies the balanced evaluation pool and token-weighted perplexity comparisons.

A.6 PARAMETER COUNTS AND ANALYTICAL COMPUTE

  • 0.62% for SmolLM2 and 0.34% for Gemma: Persistent increases analytical forward FLOPs relative to Ordinary MoE by approximately these amounts.Persistent and latest-write-only have identical parameter and analytical active-compute counts within each family; the accounting is not measured wall-clock latency.Source: §A.6 · p. 15

A.7 COMMON-HARDWARE RUNTIME MEASUREMENTS

  • Runtime differences have opposite directions across the two model families, so wall-clock measurements are descriptive implementation results rather than evidence of a general speedup or slowdown.Inference uses full length-2048 forward passes; training timing is a single-GPU forward–backward pass.Source: §A.7 · p. 16

A.8 SOURCE FREEZES AND ARTIFACT IDENTITY

  • The implementation freeze and run manifests bind source, configuration, environment, and result artifacts to recorded cryptographic identities.The seed registry also records a structured content hash; the principal architecture-level comparisons pair seeds 42, 43, and 44.Source: §A.8 · p. 16

B EVALUATION PROTOCOLS AND FUTURE-ROUTING PROBES

  • Analysis choices are frozen before final-evaluation examples are inspected, but “held out” does not mean the source documents were absent from original-model pretraining.The protocols separate construction, calibration where needed, and final evaluation roles.Source: §B.1 · p. 16, B EVALUATION PROTOCOLS AND… · p. 16

B.1 ANALYSIS ROLES AND EVALUATION POOLS

  • The language-model and local routing-state analyses use 96 length-2048 sequences per family, balanced across prose, code, and mathematics.Stable sequence identities support paired comparisons across matched architecture variants.Source: §B.1 · p. 17

B.2 VALIDATION PERPLEXITY

  • Perplexity is computed on the balanced 96-sequence pool, and absolute perplexities are not compared between families because they use different tokenizers.Training seed, rather than token count or bootstrap replicate, is the replication unit for architecture-level claims.Source: §B.2 · p. 17

B.3 DOWNSTREAM CAPABILITY EVALUATION

The appendix specifies downstream task scoring and capability metrics, then documents local routing-state suppression, future-routing probes, representation controls, persistence summaries, and ledger reconstruction checks. The reported results include local expert-set changes, persistent-state probe advantages, and a boundary on what the ledger explains.

B.3 DOWNSTREAM CAPABILITY EVALUATION

  • The evaluation covers HellaSwag, ARC-Easy, PIQA, and WinoGrande zero-shot; its macro score is the unweighted mean of four primary task metrics.The primary metric is normalized accuracy for the first three tasks and raw accuracy for WinoGrande; Table 7 reports task-level changes relative to matched Ordinary MoE, but supplied passages give no values.Source: §B.3 · pp. 17–18, Table 7 · p. 18, Table 6 · p. 18

B.4 LOCAL ROUTING-STATE SUPPRESSION

  • Local removal of the state term changes the selected Top-2 expert set in 87.6% of SmolLM2 and 69.9% of Gemma decisions.This is a local same-state counterfactual: the hidden representation and state remain fixed, and the modified selection is not propagated through later layers.Source: §B.4 · pp. 18–19

B.5 INDEPENDENT FUTURE-ROUTING PROBES

  • Independent probes predict a later router’s clean Top-1 expert from same-position representations at earlier layers, with disjoint training, calibration, and evaluation roles.The candidate representations include native routing state and model hidden-stream representations.Source: §B.5 · p. 19, §B.6 · p. 19

B.6 PROBE REPRESENTATIONS AND FIT SELECTION

  • The representation study compares native state with five hidden-stream representations, including supervised hidden subspace-128 learned from the same Persistent model.The supervised hidden subspace can match or exceed native state with sufficient probe training; these probes characterize accessibility, not an equal-resource comparison.Source: §B.6 · p. 19, §B.8 · p. 21, §B.9 · p. 21

B.7 TRANSITION-ONLY DIAGNOSTIC

  • The transition-only mask selects examples with differing sorted Top-2 numeric index sets, not semantic expert changes.Experts at different layers are distinct modules, so equal numeric indices do not imply persistence of a physical or semantic expert identity.Source: §B.7 · p. 20

B.8 PROBE AGGREGATION AND THE TWO PERSISTENCE SUMMARIES

  • At the 106-example budget, Persistent-minus-latest-write-only native-state contrasts on the Figure 3 aggregate are 22.5 pp for SmolLM2 and 10.4 pp for Gemma.These learning-curve contrasts average six source–target pairs, including horizons 1, 4, and 8; the separate pre-specified long-horizon endpoint uses only horizons 4 and 8.Source: §B.8 · pp. 20–21, Table 9 · p. 21

B.9 ADDITIONAL REPRESENTATION CONTROLS

  • The hidden-representation controls test unrestricted linear accessibility, nonlinear probe capacity, unsupervised variance, and arbitrary low-dimensional projection.They are not parameter matched to native routing state, so the comparison does not establish equal-resource performance.Source: §B.9 · p. 21

C.1 NATIVE PERSISTENT LEDGER AND NUMERICAL RECONSTRUCTION

  • The Persistent ledger sums realized post-clamp state changes to reconstruct direct state-mediated router-margin contributions, with numerical residuals more than an order of magnitude below the 10^-4 acceptance tolerance.The decomposition excludes influences transmitted through the ordinary hidden stream; finite-precision capture means algebraic exactness need not be bit-exact.Source: §C.1 · p. 22

C.2 DISTRIBUTED-HISTORY METRICS

The distributed-history analysis measures how absolute direct state-mediated routing contribution is allocated across writes, and reports that most contribution is non-recent. The appendix distinguishes Persistent’s realized-write ledger from latest-write-only’s structural audit, specifies source suppression, and qualifies the control comparison and intervention-ranking procedures.

C.2 DISTRIBUTED-HISTORY METRICS

  • Non-recent writes account for 91.7% of absolute contribution on SmolLM2 and 90.3% on Gemma; effective source support is 9.49 and 5.82 writes, respectively.These are absolute-contribution summaries, not net signed support or downstream causal effects; target layers are equally weighted within checkpoints, with seeds as replication units.Source: §C.2 · pp. 23–24

C.3 LATEST-WRITE-ONLY DIRECT-STATE AUDIT OBJECT

  • Latest-write-only assigns direct native-state contribution only to the immediately preceding replacement source; earlier sources score zero under this structural audit.Those zeros do not imply zero indirect influence through the hidden stream or earlier routing consequences.Source: §C.3 · p. 24

C.4 SOURCE-SUPPRESSION SEMANTICS

  • Full-forward suppression replaces a selected source’s post-source state with its pre-source state, then reruns later computation naturally.Later hidden states, routers, expert selections, writes, and output logits are allowed to change; for latest-write-only this skips the replacement rather than setting it to zero.Source: §C.4 · pp. 24–25

C.5 STATE GEOMETRY AND SCOPE OF THE LATEST-WRITE-ONLY CONTROL

  • The matched latest-write-only control does not isolate accumulation while controlling away state geometry and optimization differences.Persistent may accumulate to the [−5, 5] clamp, whereas latest-write-only proposals are coordinate-wise confined to [−1, 1]; magnitude, variance, and saturation frequency are not held fixed.Source: §C.5 · p. 25

D ATTRIBUTION, FULL-FORWARD INTERVENTION, AND AUDIT COST

  • The full-forward attribution study uses 24 fixed final-evaluation sequences and 96 target routing decisions per model family, selected deterministically and independently of outcomes.The appendix specifies fixed targets, source rankings, intervention metrics, and audit-cost measurements.Source: D ATTRIBUTION, FULL-FORWARD… · p. 25, §D.1 · pp. 25–26

D.1 TARGET-SEQUENCE AND DECISION SELECTION

  • The fixed target layers are SmolLM2 {15, 22} and Gemma {12, 16}, with two selected positions per sequence–target-layer pair.The resulting sequence and token identities are reused across matched variants.Source: §D.1 · p. 26

D.2 AUDITED ROUTING BOUNDARY

  • The audited boundary compares the clean second-selected expert with the clean best-excluded expert, while routing decisions remain free to change after intervention.The identities used to evaluate this scalar margin are fixed from the clean trajectory.Source: §D.2 · p. 26

D.3 SA-MOE SOURCE-RANKING METHODS

  • Source rankings include ledger magnitude, write norm, and recency; gradient×write is a local clean-trajectory sensitivity score.The sensitivity score uses the gradient with respect to the post-source state and fixed clean target experts, rather than intervention outcomes during ranking.Source: §D.3 · pp. 26–27

D.4 FULL-FORWARD SOURCE INTERVENTION AND OUTCOME METRICS

The appendix defines full-forward source interventions and their routing and output outcomes, then evaluates source-ranking quality against exhaustive intervention effects. In the fixed-checkpoint SA-MoE diagnostic, gradient×write ranks better than the ledger on the displayed metrics, while ordinary-MoE post-hoc attribution also recovers meaningful source rankings.

D.4 FULL-FORWARD SOURCE INTERVENTION AND OUTCOME METRICS

  • Full-forward source suppression allows later routing and outputs to change, and outcomes include target and downstream routing divergence, output KL, output Top-1 change, and next-token NLL effect.The audited scalar boundary holds the clean expert identities fixed while evaluating the margin after intervention.Source: §D.4 · p. 27

D.5 EXHAUSTIVE SOURCE-RANKING METRICS

  • Exhaustive ranking diagnostics compare a method’s top-ranked source with oracle intervention effects using regret, top-source accuracy, and within-target Spearman correlation.Normalized regret is defined only for targets with positive oracle effect; tied oracle-maximal sources count as correct for top-source accuracy.Source: §D.5 · p. 28, Table 11 · p. 28

D.6 FIXED-CHECKPOINT SA-MOE RANKING DIAGNOSTIC

  • In the fixed-checkpoint Persistent diagnostic, gradient×write ranks stronger than the ledger under all three displayed metrics, so exact forward contribution is not intervention sensitivity.The ledger remains informative about full-forward suppression effects on its audited boundary, but is not claimed as an optimal causal-source ranking.Source: §D.6 · p. 28, Table 11 · p. 28

D.7 ORDINARY-MOE POST-HOC ATTRIBUTION

  • Li-style post-hoc attribution recovers meaningful source rankings in Ordinary MoEs, and gradient-based sensitivity is competitive or stronger on intervention ranking.These Ordinary-MoE audit objects are residual components, so their raw intervention magnitudes are not directly comparable with SA-MoE write-intervention magnitudes.Source: §D.7 · p. 29, Table 12 · p. 29

D.8 AUDIT COST UNDER A COMMON CAPTURE POLICY

Under a common capture policy, retained SA-MoE writes support later ledger queries without replay; replicated suppression results show downstream effects but not uniformly larger effects than latest-write-only.

D.8 AUDIT COST UNDER A COMMON CAPTURE POLICY

  • Retaining every bf16 realized 128-dimensional write from a length-2048 pass requires 15.00 MiB for SmolLM2 and 9.00 MiB for Gemma.Adding clean Top-3 expert identities and logits brings the traces to approximately 15.70 MiB and 9.42 MiB, respectively; the live state alone is 0.50 MiB.Source: §D.8 · pp. 29–30
  • With the write trace retained, later ledger queries need neither another forward pass nor a target-dependent backward pass.Li-style fixed-denominator scores can also avoid replay or backward propagation when the required ordinary-MoE residual components are retained; no comparable serialized byte footprint is reported.Source: §D.8 · p. 30
  • The measurements establish an operational distinction, not a universal runtime ordering, because retention policy, component granularity, target count, and gradient requirements alter the relevant cost.Gradient×write and gradient×component require target-dependent gradient information unless a differentiable clean graph is retained, which changes the memory trade-off.Source: §D.8 · p. 30

D.9 REPLICATED DOWNSTREAM CONSEQUENCES

  • Persistent-minus-latest-write-only margin contrasts are 0.057 (95% CI [−0.213, 0.327]) for SmolLM2 and 0.113 ([−0.073, 0.299]) for Gemma.These are paired contrasts for the ledger-control advantage of Equation 47.Source: §D.9 · p. 30, Table 14 · p. 31
  • Ledger-selected Persistent writes have replicated effects on audited target margin, later routing, and output distributions, but not uniformly resolved increases over latest-write-only for every intervention outcome.The reported outcomes use three-seed means with 95% Student-t intervals (n = 3, df = 2).Source: §D.9 · p. 30, Table 14 · p. 31

E COMPLETE PER-SEED RESULTS AND ANALYSIS SCOPE

Appendix E collects seed-level values underlying the replicated analyses and clarifies metric interpretation, replication units, and the scope of uncertainty claims.

E.7 SCOPE OF STATISTICAL CLAIMS

  • Training seeds 42, 43, and 44 are the replication units where three independently trained checkpoints are available; sequence-bootstrap intervals do not substitute for training replication.Some supplementary forensic analyses use one frozen checkpoint and therefore are not three-seed architecture-level evidence.Source: E COMPLETE PER-SEED RESULTS AND… · p. 31, §E.7 · p. 33, Table 22 · p. 35

E.1 MODEL QUALITY AND DOWNSTREAM CAPABILITY

  • For Table 15, PPL is a ratio to matched Ordinary MoE, while task columns are absolute percentage-point changes; ratios above one mean higher perplexity and positive task values mean higher accuracy.PPL quantities are analyzed on the natural-log scale before transforming means and intervals back to ratios; additive quantities use the arithmetic scale.Source: §E.1 · p. 31

E.2 STATE USE AND PERSISTENCE ENDPOINTS

  • The long-horizon persistence contrast is positive in all three training seeds for both model families, while the two intervention-derived persistence contrasts have positive means but intervals spanning zero.Table 17 reports long-horizon accessibility in percentage points and intervention endpoints in target-margin units.Source: §E.2 · p. 32, Table 17 · p. 32

E.3 FUTURE-ROUTING PROBE CURVES

  • At the 106-example budget, Gemma's within-seed Persistent-minus-latest-write-only native-state gaps are 10.25, 12.55, and 8.45 percentage points.These are transition-only accuracy gaps averaged across the six frozen source–target pairs.Source: §E.3 · p. 32, Table 18 · p. 33

E.4 DISTRIBUTED PERSISTENT HISTORY

  • Under the structural direct-state audit, latest-write-only has non-recent share zero, effective support one, and largest-source share one by construction.Contribution shares use absolute selected-set-margin contribution, while signed margin is the net direct state-mediated contribution.Source: §E.4 · p. 32, Table 19 · p. 33

E.5 ORDINARY-MOE POST-HOC ATTRIBUTION ACROSS SEEDS

  • The Ordinary-MoE attribution comparison concerns residual components and is kept separate from SA-MoE write interventions; Table 20 evaluates Li Top-2-boundary adaptation and gradient×component.Higher Spearman correlation and top-source accuracy are better, while lower regret is better.Source: §E.5 · p. 33, Table 20 · p. 34

E.6 FULL-FORWARD DOWNSTREAM CONSEQUENCES

  • Persistent ledger-selected suppression has nonzero mean downstream effects in both families, but Persistent-minus-latest-write-only differences are not positive for every outcome and seed.Table 21 reports target |∆m|, downstream Top-2 mean Jaccard distance, and KL divergence between clean and intervened output distributions.Source: §E.6 · p. 33, Table 21 · p. 34

E.7 SCOPE OF STATISTICAL CLAIMS

  • Across replicated analyses, the paper reports individual training-seed values rather than treating tokens, probe examples, routing decisions, or bootstrap draws as independent model replicates.Same-sign effects across three seeds provide replication evidence but are not a separate high-powered significance test.Source: §E.7 · p. 34, Table 22 · p. 35

Related papers

Loading 2609.34634v1…