Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity

1 Introduction

• Existing approaches treat exploration as externally designed and can fail under non-stationarity or sparse, absent, or shifting rewards.The introduction also identifies premature exploitation and unstructured randomness as failure modes.Source: §1 · p. 1
• The proposed framework posits that exploration emerges when incoherence is intermediate, rather than too low or too high.The paper names this prediction the Curiosity Window conjecture.Source: §1 · pp. 1–2
• The paper implements its ideas on a Liquid State Machine substrate and compares agents with Active Inference baselines.It also tests whether endogenous noise modulation can sustain exploration without external signals.Source: §1 · p. 2
• The C2 metric peaks at intermediate noise, with > 1.5× drop-off on both sides.C2 combines coherent basins, global overlap, and transition rate.Source: §1 · p. 2
• The curiosity window is specific to the proposed model; Active Inference agents show no analogous peak.The introduction attributes this comparison to Section 5.3.Source: §1 · p. 2
• The framework validates spatial coupling, sedimentation learning, and non-stationary adaptation.These extended dynamics are listed as validated contributions.Source: §1 · p. 2

2 Summary of the Coherence Framework

• The framework makes coherence regulation between reach and yield primary, with representational structures treated as secondary.Its principles require functional differentiation, mutual constraint, and memory to affect the trajectory of reach and yield.Source: §2 · p. 2

2.1 Mathematical Setting

• Reach represents viable continuations, yield represents environmental and internal constraint, and memory represents sedimented traces of past coherence.The global overlap between reach and yield equals 1 when the distributions match and approaches 0 when their supports are disjoint.Source: §2.1 · pp. 2–3
• The Curiosity Window conjecture predicts maximal curiosity at intermediate incoherence, with trapping below and fragmentation above that regime.The framework associates curiosity with distinct coherent configurations, global integration, and structured transitions between basins.Source: §2.1 · p. 3

2.2 Representation Theorem

• The representation theorem states that a continuous, separately monotone curiosity functional satisfying the stated boundary conditions is unique up to monotone transformation in product form.The product combines global overlap, coherent basin count, and inter-basin transition rate.Source: §2.2 · p. 3

2.3 Sedimentation as Learning

• Learning is modeled as sedimentation: repeated low-incoherence states gain probability mass in the baseline reach, rather than changing parameters through a loss function.The sedimentation rate and recency weights parameterize the process.Source: §2.3 · p. 3

3 Computational Architecture and Implementation

• The architecture separates a high-dimensional dynamical substrate from a control layer that regulates coherence and exploration.The LSM supplies a fading-memory representation, while the coherence layer governs organization and exploration.Source: §3 · p. 4

3.1 Liquid State Machine Substrate

• The LSM acts as a dynamical medium, providing nonlinear input expansion, fading memory, and temporally rich states over which coherence variables evolve.The reservoir itself is not the agent.Source: §3.1 · p. 4

3.2 State Augmentation and Temporal Thickness

• A slow trace augments the instantaneous reservoir state with sedimented temporal structure.The resulting representation combines fast dynamics with the slow trace.Source: §3.2 · p. 4

3.3 Field Representation and Extraction

• The reach, yield, and memory fields are represented as probability distributions over a discretized reservoir state space.Yield reflects environmental and internal constraint, while memory summarizes past coherent states.Source: §3.3 · pp. 4–5

3.4 Field Dynamics

• The field dynamics couple reach and yield, integrate past reach states into memory, and continually reconfigure rather than converge to a static optimum.Memory is a slow-moving summary of recent and recurrent interactions, not a replay buffer or explicit transition store.Source: §3.4 · p. 5

3.5 Endogenous Exploration Mechanism

• Endogenous modulation replaces externally imposed noise, producing low noise at high coherence and extreme incoherence and maximal exploration at intermediate incoherence.This closes a feedback loop between internal state and exploration.Source: §3.5 · p. 5

3.6 Spatial Structure and Basin Dynamics

• Spatial coupling lets local perturbations propagate across the field, while sedimentation implements learning as structural bias.The resulting exploration is generated by the system’s coherence dynamics rather than injected from outside.Source: §3.6 · p. 6

4 The Curiosity Window

• Effective exploration is predicted to occur within a bounded intermediate incoherence regime, between behavioral trapping and fragmented, unproductive exploration.The intermediate regime is described as allowing multiple possibilities while retaining enough coherence to make exploration meaningful.Source: §4 · p. 6
• The transition-rate term distinguishes active movement between basins from merely possessing the potential to move between them.Global overlap supports integration, while coherent basin count represents differentiated metastable structure.Source: §4 · pp. 6–7
• C2 multiplies global overlap, coherent basin count, and inter-basin transition rate to measure integrated, differentiated, dynamically reorganizing behavior.The framework presents this functional as uniquely determined up to monotone transformation under its stated conditions.Source: §4 · p. 7

5.1 Simulation 1: Phase-Diagram Sweep

• In a five-basin, 40-cell landscape, the study used 45 logarithmically spaced values and measured incoherence, global overlap, exploration entropy, basin dwell time, coherent-basin count, and transition rate.Field-coupling strength was fixed at 0.3, and each run lasted 1,500 time steps.Source: §5 · p. 7, §5.1 · p. 7

5.2 Simulation 2: Three-Regime Time Series

• C2 has a distinct interior peak, with a curiosity window about 1.74 decades wide in log-space.The peak reflects structured exploration rather than entropy alone.Source: Figure 1 · p. 8, §5.2 · p. 8
• Low noise traps the system in one basin, while high noise fragments transitions; only intermediate noise supports coherent exploration across multiple basins.The three-regime pattern pairs high overlap with trapping at low noise, and collapsed overlap with frequent transitions at high noise.Source: §5.2 · p. 8, Figure 2 · p. 9

5.3 Endogenous Exploration

• With constant noise, setting σπ to zero cuts exploration to 1% of its pre-cutoff rate; endogenous σπ(t) = ψ(I(t))·G(t) is proposed as a self-sustaining feedback loop.The described loop links incoherence to increased exploration and renewed coherence.Source: §5.3 · p. 9

5.4 Extended Dynamics

• Spatial coupling spreads local disturbances across the field within 5–10 timesteps; sedimentation improves convergence and final coherence, while endogenous ψ(I) recovers fastest after basin shifts.The sedimentation comparison used α = 0.3 across 50 episodes.Source: §5.4 · pp. 9–10

5.5 Sedimentation as Structural Learning

• Sedimentation deforms the reach probability landscape rather than storing facts, biasing future coherence-seeking toward previously successful regions.In the 60-episode run, memory entropy fell by 0.075 nats, and the top 10 field positions held 33.2% of probability mass.Source: §5.5 · p. 10, §5.5.2 · p. 12

5.5.3 Sedimentation Tracks Environmental Relevance

• As reward emphasis shifted from Basin 0 to Basin 2, sedimentation tracked relevance: Basin 2 ended with the largest memory mass, while older patterns faded.Final masses were 0.376 for Basin 2, 0.290 for Basin 1, and 0.194 for Basin 0.Source: §5.5.3 · p. 12

5.5.4 Sedimentation Confers a Measurable Learning Advantage

• The memory agent outperformed the memoryless control on coherence-related metrics, with a +5.9 percentage-point advantage in incoherence reduction.The reported mechanism is an additional pull toward previously successful attractor configurations.Source: Table 1 · p. 12, §5.5.4 · p. 12

6.1 ECF–PPO Hybrid Approach

• ECF–PPO retains PPO’s policy and value updates while augmenting its reward with an intrinsic signal derived from internal consistency, novelty, and memory mismatch.Its internal expectation, current experience, and sedimented memory guide structured exploration.Source: §6.1 · pp. 13–14

6.2.1 ECF-RichMem Algorithm

• ECF-RichMem extends ECF with fast, medium, and slow memory components, distinguishing transient fluctuations from persistent environmental changes.The architecture derives intrinsic reward from incoherence and novelty relative to memory, with coherence gating.Source: §6.2.1 · pp. 14–16, Figure 5 · p. 15

6.2.2 Experimental Protocol

• The LunarLander-v2 protocol separates baseline learning, retention, wind disturbance, and recovery across 500 epochs and 10 seeds.Performance is average episodic return over the final portion of each phase.Source: §6.2.2 · p. 16
• ECF-RichMem led LunarLander-v2 across all four phases, with means of 16.3 in Phase 1, 40.0 in Phase 2, −16.7 in Phase 3, and 5.7 in Phase 4.The results attribute its retention and disturbance profile to multi-timescale memory; Table 2 reports mean ± standard deviation across seeds.Source: §6.2.2 · pp. 16–17, Table 2 · p. 16

6.3 BipedalWalker-v3 Continuous Control

• BipedalWalker-v3 tests continuous joint-torque control using PPO and PPO+ECF under training, reward-free retention, gravity disturbance, and recovery.The comparison evaluates whether ECF adds value on top of PPO.Source: §6.3 · pp. 17–18
• PPO+ECF and PPO were similar in stationary BipedalWalker-v3 phases, but under gravity disturbance their means were 45.3 and −20.3, respectively.PPO+ECF retained a slight Phase 4 advantage, 5.4 versus 3.1, with high variance; neither method fully recovered.Source: §6.3 · p. 18, Table 3 · p. 18

6.4 Cross-Experiment Analysis

• Under disturbance, ECF-RichMem outperformed ECF and ICM in LunarLander-v2, while PPO+ECF maintained a positive mean as PPO fell to −20.3 in BipedalWalker-v3.The cross-experiment analysis reports the largest ECF advantage under non-stationarity or absent reward.Source: §6.2.2 · p. 16, Table 2 · p. 16, §6.4 · p. 18
• Across experiments, sedimented memory peaks in frequently visited, high-coherence regions and persists through environmental changes; constant noise yielded either excessive or insufficient exploration.The reported advantage is smallest in stable, rewarded settings and largest under non-stationarity or absent reward.Source: §6.4 · p. 18

7.1 Main Differences of the ECF Approach

• ECF tracks expectation, current experience, and sedimented memory, using their relationships to distinguish familiar patterns, transient fluctuations, and meaningful deviations.The discussion presents memory-guided adaptation, internal learning pressure under weak or absent rewards, transfer to related settings, and inspectable internal signals as potential advantages.Source: §7.1 · p. 19

7.2 Relationship to Existing Frameworks

• The proposed feedback loop couples incoherence-driven exploration to changes in the coherence landscape; simulations report that endogenous exploration fails without ψ(I).The discussion frames this as a falsifiable distinction from systems lacking the endogenous modulation.Source: §7.2 · p. 19

7.3 Limitations

• The reported limitations include untested scaling beyond 50–60 reservoir nodes, short basin dwell times, metric sensitivity, high PPO+ECF variance, and no deep-policy integration beyond PPO.Only C2 clearly supports the conjecture; alternative metrics show weaker effects. BipedalWalker PPO+ECF results show substantially higher variance than PPO.Source: §7.3 · p. 19

7.4 Hardware Substrate Considerations

• The substrate evaluation scored a neuromorphic reservoir with digital coherence control highest, at 33/40.The discussion describes neuromorphic hardware as a fit for the LSM reservoir and optical hardware as a potential route to faster reservoir computation.Source: §7.4 · p. 20

• The Curiosity Window is confirmed under C2, absent in Active Inference agents, and self-sustaining with endogenous ψ(I) modulation.Source: §8 · p. 20
• Endogenous σπ(t) = ψ(I(t)) · G(t) modulation accompanies particular strength under non-stationarity and environmental disturbance in RL experiments.Source: §8 · p. 20
• C2 consistently shows a clear interior peak across parameter sweeps; future work should address higher-dimensional scaling, broader policy architectures, and richer environments.Source: §8 · p. 20

A Analytical Proof of the Curiosity Window

• The conjecture predicts sustained intrinsic curiosity only within incoherence thresholds, proved in closed form for two basins with bounds and an extension sketch for K > 2.Source: A Analytical Proof of the Curiosity… · p. 21

A.1 The 2-Basin System

• The two-basin setup fixes a preferred yield q > 1/2 and models reach as a scalar under frozen-yield mirror flow with additive Gaussian noise.Source: §A.1 · p. 21

A.2 Deterministic Dynamics

• Without noise, the mirror flow converges uniquely to basin alignment at p∗ = q, where incoherence is zero and overlap is one; exploration then stops.Source: §A.2 · p. 21

A.3 Stochastic Dynamics and the Effective Potential

• The noisy analysis characterizes the stationary density and effective inverse temperature, with a barrier separating the preferred attractor from the opposite basin.Source: §A.3 · p. 22

A.4 Three Regimes

• At low noise the system is locked; at high noise it fragments, while intermediate noise permits basin visits without losing coherent occupancy.Source: §A.4 · p. 22
• C2 vanishes as noise approaches zero or infinity and attains a unique interior maximum at σ∗.Source: §A.4 · p. 23

A.5 Explicit Window Bounds

• The explicit window is defined by a threshold on C2 and translated into bounds on expected incoherence using a Laplace relation.Source: §A.5 · p. 24

• The curiosity window and unique optimum persist under basin asymmetry; higher barriers require more noise to escape, while the overlap penalty also grows.Source: §A.6 · p. 24

• The proof sets C=0 when T=0, G is below its coherence threshold, or B=0, citing absent transitions or fragmented dynamics.Source: B Proof of Theorem 1 · p. 24
• For c>0, f=F on the positive orthant has smooth level surfaces τ=h_c(g,b), decreasing in g and b and asymptotic to coordinate planes.Source: B Proof of Theorem 1 · p. 24
• Dimensional consistency requires F to be invariant under independent rescaling of its dimensionless arguments, which are measured on different scales.Source: B Proof of Theorem 1 · p. 24
• Dimensional consistency and the multiplicative Cauchy theorem yield C=φ(G·B·T), with positive exponents and C>0.Source: B Proof of Theorem 1 · p. 24
• Any other admissible C is a monotone transformation of C2 and therefore preserves the ordering over system states.Source: B Proof of Theorem 1 · p. 25