Self-discovering RL in the Era of Experience: Is Learning History an Asset or a Burden?

1 INTRODUCTION

• The audit targets a mechanistic gap: Disco103’s persistent recurrent state carries history across learner updates, but its use under changing conditions had not been inspected.The introduction situates the question in the debate between LLM-based self-improvement and learning from open-ended interaction.Source: §1 · pp. 1–2
• The audit operationalizes the Era of Experience as five demands: horizon, reward units, streams, change, and exploration.These test extended competence, numerical robustness, continuity, adaptation under task shifts, and discovery versus consolidation.Source: §1 · p. 2
• Zero-state pinning, timed freezing, natural transplantation, and perpetual clamping provide surgical interventions into recurrent history.The work presents the resulting capability profile and numerical-interface analysis as a repeatable audit standard for self-evolving RL algorithms.Source: §1 · p. 2
• A live recurrent state sustains a six-decade usable reward-scale window on Catch, versus three decades under zero-pinning.At normalized score threshold S ≥0.8, the gap is 3 decades, with multiplicity-adjusted interval [2, 3].Source: §1 · p. 2
• Perpetual clamping amplifies the penalty of mismatched recurrent-state transfer at low recipient scale, while natural evolution attenuates it.The reported interaction is 0.393 [0.079, 0.861].Source: §1 · p. 2
• Matching replay capacities reverses Disco103’s apparent recovery advantage over DQN under Catch action reversal, implicating external data turnover in adaptation comparisons.The authors use controlled environments to isolate memory inheritance and replay turnover from high-dimensional perception.Source: §1 · p. 2

2 EXPERIMENTAL ACCESS TO LEARNING HISTORY

• The learner update rule produces an update and next recurrent and persistent states, after which an optimizer updates agent parameters.Rule weights η remain fixed while agent parameters and recurrent history evolve.Source: §2 · p. 2, Figure 1 · p. 3
• The audit presents five distinct capability axes, each with its own question, readout, and explanatory control.Implementation checks report maximum relative forward discrepancy of 1.597 × 10−5 and maximum absolute post-step parameter discrepancy of 2.733 × 10−4 against JAX.Source: Table 1 · p. 3, §2 · pp. 3–4
• Disco103’s recurrent state is the lifetime meta-RNN pair (h, c), distinct from retained optimizer moments and other persistent quantities.The sampled experience and predictions depend on the current policy and replay; moving averages and target parameters are also persistent.Source: §2 · p. 3
• Zero-pinning, timed freezing, and transplantation vary recurrent-state availability, accumulation time, and origin while within-iteration recurrence and agent updates remain active.Natural transfer initializes a recipient once with donor state; perpetual clamping restores the donor state after every iteration.Source: §2 · p. 3, Figure 2 · p. 4
• The scale window is the widest contiguous tested log-scale interval that meets a fixed capability or relative-retention threshold.A capable reference agent supplies the behavioral basis for interpreting state interventions.Source: §2 · p. 4

3 THE NUMERICAL GEOMETRY OF A SELF-DISCOVERED UPDATE

• The 601-atom value head transforms and clips scalar targets, projects them onto neighboring atoms, and applies the inverse transform after expectation for scalar readout.For in-support targets, two-hot interpolation preserves the transformed first moment, allowing inverse readout to recover the target in exact arithmetic.Source: §3 · pp. 4–5
• At unit scale, default, linear, and narrow-support interfaces score 0.997, 0.999, and 1.000, while effective-bin statistics are 2.22, 1.47, and 2.10, respectively.Thus similar behavior coexists with different internal categorical representations; effective bins summarize aggregate categorical use.Source: §3 · p. 5
• At scale 10−3, live-minus-zero state effects are 0.822, 0.925, and 0.962 for default, linear, and narrow-support interfaces, with all family-adjusted intervals above zero.Interaction intervals span zero, leaving differences in effect magnitude across interfaces unresolved.Source: §3 · p. 5
• Changing the value interface changes categorical supervision while preserving the logit-gradient form of the value loss.The scalar target is detached, and the cross-scale assay tests whether state-supported learning depends on a particular interface.Source: §3 · p. 5

4.1 A1: LEARNING HORIZON SEPARATES PACE FROM ATTAINED COMPETENCE

• At learning rate 10−2, Disco reaches CatchBig endpoint 0.9739 and its own attained-level target around 133,400 interactions; at 3 × 10−4, it reaches 0.9948 after 1.16 million interactions, mastering around 500,200.At 10−2, freezing at iteration 400 yields 0.9513 versus 0.7896 under zero-pinning.Source: §4.1 · p. 5

4.2 A2: REWARD UNITS EXPOSE STATE SUPPORT AND REVISABLE INHERITANCE

• Live state supports a six-decade usable reward-scale window on Catch, versus three decades under zero-pinning; CatchDense shows the same pattern, while CatchLong shows three versus two.At low recipient scale, cross-scale donors yield 0.5307 under natural transfer versus 0.0165 under repeated clamping; the mismatch-by-inheritance interaction is 0.39265 [0.07896, 0.86094].Source: §4.2 · p. 5, Figure 3 · p. 6

4.3 A3: CONTINUING DYNAMICS DIFFER FROM MISSING BOUNDARY SIGNALS

• In CatchStream, Disco’s normalized reward rate is 0.939 versus DQN’s 0.769, while Tracking reverses that order, with DQN at 0.964 and Disco at 0.731.For A2C, masking termination in episodic Catch drops the score from 0.99615 to 0.00070186, while supplying a synthetic signal every 29 steps restores it to 0.99231; physical resets remain.Source: §4.3 · p. 6

4.4 A4: ENVIRONMENTAL CHANGE TESTS THE RELEVANCE OF RETAINED EXPERIENCE

• Under Breakout action reversal, Disco’s return changes from 9.804 before the switch to 8.763 at the end, while A2C and PPO end at 5.218 and 5.321.With replay controlled, Disco-minus-DQN post-switch AUC changes from +0.516 with the original DQN buffer to −0.090 with matched capacity and −0.315 after switch-time clearing; adjusted intervals exclude zero.Source: §4.4 · pp. 6–7, Figure 4 · p. 7

4.5 A5: EXPLORATION SEPARATES INFORMATIVE EXPERIENCE FROM CONSOLIDATION

• At DeepSea depth 12, Disco, A2C, and DQN reach the maximum tested majority-solved depth, solving 12/15, 11/15, and 11/15 runs; PPO reaches majority depth 8 and Disco’s entropy ablation depth 10.Reward discovery does not ensure reliable behavior: at depth 14, A2C discovers reward in 12/15 runs but has aggregated final-window success zero.Source: §4.5 · pp. 7–8

5 WHAT MUST A SELF-DISCOVERED RULE RETAIN?

• At low recipient scale, allowing imported history to evolve attenuates the donor-mismatch penalty relative to perpetual clamping, motivating revisable rather than immutable retention.The mismatch-by-inheritance interaction is 0.393 [0.079, 0.861].Source: §5 · p. 8
• Matching replay capacities reverses the apparent recovery advantage over DQN, so internal recurrent state and external replay retention should be independently controllable.The paper proposes measuring their roles and timescales separately.Source: §5 · p. 8

5.1 CAPABILITY GATES AND DIAGNOSTIC MODEL SYSTEMS

• All 30 full-Atari Pong runs remain near the losing floor, and a fixed 18-observation probe at one million interactions shows zero first-layer activations; diagnose representation and optimization before memory-supported competence.The stated capability gate limits when memory effects can be meaningfully audited.Source: §5.1 · p. 8
• The audit protocol transfers to OPEN, where one of six endpoint live-minus-zero contrasts has an adjusted interval above zero; future audits should establish competence before dissecting learning dynamics.The passage describes protocol portability with condition-dependent effects.Source: §5.1 · p. 8

From self-discovering algorithms to auditing their mechanics

• Prior analyses cover learned predictions and bootstrapping in Disco57, while this audit tests what Disco103’s persistent learning history does across changing conditions.The paper contrasts this with DPO, whose update has no persistent recurrent state.Source: §6 · p. 8

Scale robustness: recurrent state versus numerical geometry

• The framework separates recurrent-state and EMA interventions on scale support from interface manipulations and categorical-bin assays of numerical geometry.The inheritance factorial tests whether mismatched history’s cost depends on revisability.Source: §6 · pp. 8–9

Continual plasticity, memory hierarchy, and data retention

• Replay can couple stored trajectories to hidden-state reconstruction, and the paper’s replay-controlled assays show data retention can confound attribution of recovery to internal plasticity.Its evaluation also uses controlled task transformations, internal-state interventions, seed-block bootstrap intervals, and cross-framework numerical verification.Source: §6 · p. 9

7 CONCLUSION

• The five-axis profile distinguishes attained competence from learning pace, continuous interaction from boundary information, and reward discovery from consolidation.The audit operationalizes the Era of Experience to examine self-discovered reinforcement learning as a causal system.Source: §7 · p. 9
• History supports learning across reward scales, but mismatched history becomes a greater burden when revision is blocked at low recipient scale.Replay controls further separate apparent adaptation from internal plasticity by revealing the contribution of external data turnover.Source: §7 · p. 9
• The protocol is presented as an explanatory blueprint and evaluation standard for next-generation self-improving learning systems.The conclusion identifies revisability as a central design principle within the audit’s scope.Source: §7 · p. 9

• AI tools assisted with manuscript and analysis tasks; the authors checked quantitative claims and figures and retain responsibility for interpretation and submitted content.Source: AI USE STATEMENT · p. 10

• The study uses no human participant data and reports experimental conditions and uncertainty to support interpretation and reproducibility.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The supplement records the five-axis experimental atlas, state operations, numerical calculations, source and runtime identities, seeds, budgets, estimands, intervals, and analysis scripts.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 AGENT AND RULE INTERFACES

• The Disco rule separates a 128-unit lifetime meta-RNN from a 256-unit trajectory RNN, and the state interventions modify lifetime state while leaving within-iteration recurrence active.The rule’s lifetime pair persists across learner iterations; transplantation transfers that pair only.Source: §A.1 · p. 13, Table 2 · p. 13

A.2 COHORTS AND INTERACTION BUDGETS

• The cohorts specify interaction budgets, seeds, task modes, replay configurations, and hardware; the reported cohort budgets are 116,928,000 E1, 22,272,000 E2, and 84,564,000 E3 interactions.The released Disco103 checkpoint is evaluated with controlled interventions; its original self-discovery compute is reported separately from evaluation cost.Source: §A.2 · pp. 13–14

A.3 EXECUTION IDENTITY AND STATE LABELS

• Each result is tied to its task, execution convention, summary window, and distinct state-condition labels.Scale windows aggregate score curves before finding the widest contiguous passing interval on the tested grid, without interpolation.Source: §A.3 · p. 14, §B.1 · p. 14

B.1 NORMALIZATION AND AGGREGATION

• Reward normalization removes scaling once, while seed aggregation uses named trimmed-mean and IQM estimators whose boundary weights differ at 18 seeds.At 12 seeds, the integer-trimmed mean and exact middle-50% IQM coincide.Source: §B.1 · p. 14

B.2 BOOTSTRAP FAMILIES

• Bootstrap resampling preserves seed-linked conditions, and family-11 intervals adjust for the planned family of contrasts.Recovery analyses use a fixed switch origin; non-attaining runs are censored at the horizon, and threshold crossing does not guarantee sustained returns.Source: §B.2 · p. 15, §B.3 · p. 15

C.1 NUMERICAL CHECKS AND INDEX CONVENTIONS

• Shared-input checks report a maximum relative forward discrepancy of 1.597 × 10^-5 and post-step parameter agreement within absolute error 2.733 × 10^-4.Scale, inheritance, Catch adaptation, and learning-rate assays use reference indexing; the paired convention sensitivity comparison is reported separately.Source: §C.1 · pp. 15–16, Table 3 · p. 15

C.2 TRANSFORMED TARGETS AND SUPPORT

• The value interface transforms scalar targets, clips them to categorical support, and linearly projects them onto neighboring atoms; for target 1000, transformed support stays in range while linear support [−300, 300] does not.The support has 601 points with default M = 300.Source: §C.2 · p. 16

E.1 INTERVENTIONS, LINEAGE, AND INDEPENDENT EVALUATIONS

• All six OPEN game–scale cells pass both capability comparisons, while only Breakout at scale 10 has a positive endpoint live-minus-zero effect with an adjusted interval above zero.The other five endpoint intervals and all four scale-interaction intervals span zero; the protocol identifies a local carry effect, not a uniform one.Source: §E.1 · p. 20

E.2 TERMINAL EVALUATION AND INFERENTIAL SCOPE

• Terminal OPEN results cover 28 estimable contrasts while retaining the original 34-contrast correction family; six planned early-reset contrasts lack required intermediate states.Terminal reset-minus-live comparisons measure endpoint sensitivity, not recovery over time; further rule comparisons can use independent checkpoints with matched capability and implementation controls.Source: §E.2 · p. 20, Table 8 · p. 20

F FUTURE-WORK FOUNDATION: COMPETENT FULL-ATARI AGENTS

• None of 30 limited-budget Pong runs improved by two raw game points, and a fixed probe found zero first-layer activations at one million interactions.The authors call for representation and optimization checks, then capability qualification, before interpreting memory effects in full Atari.Source: F FUTURE-WORK FOUNDATION: COMPETENT… · pp. 20–21

G CROSS-SYSTEM CONTEXT FOR FUTURE RULE COMPARISONS

• At equal interaction budgets, system rankings vary by task and scale: Disco leads on several Breakout cells, while OPEN and Adam lead on several Freeway cells.This comparison does not isolate the learned rule from network and update-schedule differences; within-system OPEN carry interventions address a different question.Source: G CROSS-SYSTEM CONTEXT FOR FUTURE RULE… · p. 21, Table 10 · p. 22

H.1 DEEPSEA: FIRST DISCOVERY AND SUBSEQUENT CONSOLIDATION

• At depth 14, A2C reaches the goal in 12/15 runs but has zero final-window success, showing that discovery alone does not establish sustained competence.At depth 16, every learner has zero aggregated tail success despite differing discovery counts; Table 11 also reports random-policy campaigns as opportunities for a first encounter.Source: §H.1 · p. 21, Table 11 · p. 22

H.2 HORIZON AND OPTIMIZER SETTING

• At learning rate 10−2, Disco reaches its attained-level target in 133,400 interactions; at 3 × 10−4 it reaches a slightly higher endpoint over a longer trajectory.The assay treats learning pace and endpoint competence as paired, non-interchangeable readouts.Source: §H.2 · p. 22

H.3 STREAMS DISTINGUISH BOUNDARY INFORMATION FROM PHYSICAL CONTINUITY

• CatchStream final reward-rate scores are 0.93896 for Disco, 0.76918 for DQN, and 0.00622 for A2C, whereas Tracking favors DQN over Disco on normalized rate.The task-dependent ordering motivates separating physical continuity from boundary-information effects; in MinAtar Breakout, Disco’s masked-minus-unmasked effect is −0.0042 raw reward per interaction.Source: §H.3 · p. 23

I.1 INTERFACE-BY-STATE EFFECTS; I.2 NORMALIZATION AND STATE-DIRECTION CONTROLS

• At scale 10−3, live-minus-zero tail-score differences are positive under all three tested value interfaces, while interface-by-state interaction intervals span zero.The held-out-direction assay found all eight specified contrasts had pointwise intervals spanning zero, so direction-level causal attribution remains open; the audit’s demonstrated state effects use whole-state interventions.Source: §I.1 · p. 23, Table 12 · p. 23, §I.2 · p. 24

L EXPERIMENTAL DESIGN AND CONFIGURATIONS

• The five axes vary learning horizon, reward scale and normalization, termination or continuing dynamics, within-run task mappings, and sparse-reward depth.Component interventions connect these responses to recurrent history, running normalization, and auxiliary losses; Table 14 gives the reference-indexed matrices.Source: L EXPERIMENTAL DESIGN AND CONFIGURATIONS · p. 25, Table 14 · p. 25

M.1 SCORE AND WITHIN-RUN SUMMARIES

• Episodic tasks are summarized by return and continuing tasks by reward rate, with environment-specific random and optimal references.The normalization applies each task’s references; Tracking’s random reward rate is 1/8.Source: §M.1 · p. 26
• Within-run summaries use different specified windows across assays, preserving the distinction between tail averages and endpoint observations.Catch adaptation also reports post-switch AUC and common-capability attainment, alongside a final window or endpoint.Source: §M.1 · p. 26, §M.3 · p. 27

M.2 ACROSS-SEED ESTIMATORS

• Across-seed intervals use assay-specific resampling and estimator rules, with cells below three finite values reporting their observed range instead of a bootstrap interval.Paired effects aggregate within-seed differences; attainment-time summaries retain every run through explicit censoring.Source: §M.2 · p. 26

M.3 ADAPTATION AND MASTERY

• Adaptation AUC averages performance over the post-switch horizon, while lifetime mastery records the first logged step reaching a run-specific target.The common endpoint and own-reference crossing time describe complementary learning-curve features.Source: §M.3 · p. 27

N LONG-RUN LEARNING AND OPTIMIZER SENSITIVITY

• 0.01217 [0.00170, 0.06261] is the live-minus-frozen-at-400 endpoint effect in CatchBig; the 3 × 10−4 Disco arm reaches 0.99478.E-LR selected tail IQMs of 0.97910 for A2C at 10−3, 0.91301 for DQN at 10−4, and 0.88455 for PPO at 3 × 10−4.Source: N LONG-RUN LEARNING AND OPTIMIZER… · p. 27, Table 15 · p. 27, Figure 8 · p. 28

O PERSISTENT LEARNING STATE ACROSS REWARD SCALES

• At score 0.8, live Disco spans six, six, and three reward-scale decades on Catch, CatchDense, and CatchLong, versus three, three, and two with pinned state.The window is the widest contiguous passing interval on the tested multiplier grid; CatchLong does not regain Catch’s low-scale capability.Source: O PERSISTENT LEARNING STATE ACROSS… · p. 28, Figure 9 · p. 29

O.1 EXPLICIT NORMALIZATION AND THE MINATAR COMPARISON

• At reward scale 103, PopArt scores 0.99581 versus A2C’s 0.00229, while both remain near random at 10−3.In Breakout, live and pinned Disco each retain all five tested scales; the reported state effect varies across tasks and performance criteria.Source: §O.1 · p. 33, Table 25 · p. 33

P TRANSPORTING PERSISTENT STATE BETWEEN REWARD SCALES

• At recipient scale 10−3, cross-scale state scores 0.53070 under natural transfer versus 0.01651 under repeated clamping; the clamping mismatch interaction is 0.39265 [0.07896, 0.86094].At recipient scale 103, the interaction is 0.00236 [−0.00783, 0.00889]; natural transfer permits state evolution, whereas clamping restores it after each iteration.Source: P TRANSPORTING PERSISTENT STATE… · p. 35, Table 27 · p. 35

Q THE CATEGORICAL VALUE INTERFACE

• The value interface transforms and clips scalar targets, projects them onto adjacent categorical support atoms, and reads predictions by inverting the expected transformed coordinate.The illustrated projection of 0.35 assigns weights 0.65 and 0.35 to atoms 0 and 1, preserving the first moment.Source: Figure 10 · p. 36, §Q.2 · p. 37, §Q.3 · p. 37

Q.3 PREDICTION READOUT, SCALAR TARGET, AND VALUE LOSS

• The value loss routes the runtime temporal-difference quantity into categorical supervision using the projected target and configured value-loss coefficient.The JAX path stops the gradient at the scalar target; the PyTorch path detaches the TD input and projected target.Source: §Q.3 · p. 38

Q.4 THE THREE EXECUTED INTERFACE SETTINGS

• 0.99695, 0.99949, and 0.99962 are the default, linear, and narrow-support Catch scores; their effective-bin IQMs are 2.22209, 1.46921, and 2.09795.The interface changes categorical use while performance remains close to Catch’s attainable score.Source: Figure 10 · p. 36, §Q.4 · p. 38, Table 28 · p. 39

Q.5 EFFECTIVE-BIN STATISTIC AND AGGREGATION ORDER

• Effective bins measure the concentration of the mean categorical prediction, averaging distributions across logits rows before calculating entropy.For each seed, the reported diagnostic averages the final 15% of recorded values before seed-level IQM aggregation.Source: §Q.5 · p. 38

R TERMINATION SIGNALS, RESETS, AND CONTINUING DYNAMICS

• On Tracking, DQN reaches normalized rate 0.96409 versus Disco’s 0.73087; their threshold-reaching fractions are 1 and 0.5.Tracking evolves continuously, while the comparison also includes episodic and masked Catch exposures.Source: R TERMINATION SIGNALS, RESETS, AND… · p. 39, Figure 11 · p. 40
• 0.93896, 0.00622, and 0.76918 are the CatchStream tail scores for Disco, A2C, and DQN; all Disco and DQN runs reach 0.8 at some logged time.CatchStream has continuous dynamics and a persistent paddle, with termination remaining zero.Source: Figure 11 · p. 40, R TERMINATION SIGNALS, RESETS, AND… · p. 41, Table 31 · p. 42
• In Breakout, Disco’s paired raw reward-rate difference under terminal masking is −0.0042 [−0.0056, −0.0011].The masking intervention preserves the game while changing terminal information.Source: Figure 11 · p. 40, R TERMINATION SIGNALS, RESETS, AND… · p. 42

S WITHIN-LIFETIME CHANGES AND RECOVERY MECHANISMS

• Under Breakout action reversal, Disco’s pre-switch and endpoint returns are 9.804 and 8.763; A2C and PPO end at 5.218 and 5.321.The returns are absolute episodic scores; the action-flip comparison also includes frozen-state Disco and a stationary control.Source: Figure 12 · p. 43, S WITHIN-LIFETIME CHANGES AND RECOVERY… · p. 46, Table 38 · p. 46
• Under Catch observation reversal, endpoint scores are 0.9924 for live Disco, 1.0000 for frozen Disco, and 0.3257 for A2C; under reward-sign reversal, they are 1.0000, 1.0000, and 0.9981.Frozen-state learners continue updating parameters, so frozen recurrent state is not a frozen learner.Source: S WITHIN-LIFETIME CHANGES AND RECOVERY… · pp. 42, 46, Table 39 · p. 46, Figure 14 · p. 47

T DIRECTED EXPLORATION AND THE STAIRCASE PROTOCOL

• At depth 12, reference-indexed Disco, A2C, and DQN solve 12/15, 11/15, and 11/15 runs; Disco’s deepest majority-solved depth is 12.Disco solves 2/15 at depth 14 and 0/15 at depth 16; counts beyond the majority boundary are sparse successes within the finite budget.Source: T DIRECTED EXPLORATION AND THE… · p. 47, Table 40 · p. 48, Figure 15 · p. 50

U INTEGRATED VIEWS OF STATE AND BEHAVIOR

• The integrated figures connect value-interface behavior with scale, state inheritance, replay, continuing-stream, and exploration measurements.Their panels retain task-specific score definitions.Source: U INTEGRATED VIEWS OF STATE AND BEHAVIOR · p. 50, Figure 16 · p. 51, Figure 17 · p. 52

V REPRODUCTION

• The delivered archive rebuilds the document and supports analysis reproduction, but end-to-end experiment replication additionally requires training and per-run data artifacts.The training implementation, released rule checkpoint, and task/runtime configurations define the learning runs.Source: V REPRODUCTION · pp. 51–52