PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifiability of a dissipative knowledge-dynamics model: exact recovery under designed excitation, degeneration on observational data

Arman Kostanian, Armen Beklaryan

arXiv:2610.09889v1cs.LGmath.OCstat.ML

TL;DR

Mechanistically named learning parameters are meaningful only if observations determine them, and that recoverability depends on how practice excites the system. The paper analyzes a dissipative knowledge-dynamics model and finds exact recovery under designed excitation but degeneration on observational benchmarks.

  • Problem

    Whether mechanistically named learning parameters are uniquely determined by observations depends on how the system was excited while observed.

    Source: §1 · p. 2

  • Method

    The paper analyzes a dissipative knowledge-dynamics ODE and proves conditional structural identifiability with constructive recovery for the two-concept case.

    Source: §4.5 · p. 11, §9 · p. 18

  • Results

    Spearman ρ = 0.83 between recovered forgetting rates and topic difficulty is refuted by controls showing temporal-shuffle survival, a matched Bayesian baseline, and insensitivity to real timestamps.

    Source: §1 · p. 3, §9 · p. 18

  • Takeaways & Limitations

    Designed excitation permits recovery, whereas standard observational data can make a forgetting parameter encode per-topic response frequency.

    Source: §9 · p. 18

    The theorem is constructive for K = 2; the general-K case remains open, and the stochastic term is not estimated.

    Source: §8 · p. 17

Abstract

from arXiv · show

Human learning is a dissipative dynamical process: mastery accumulates through practice, decays through forgetting, and propagates across interdependent concepts. We model it as a nonlinear dissipative system of ordinary differential equations whose parameters are mechanistically meaningful (a concept-transfer matrix encoding prerequisite coupling, per-concept forgetting rates, and a saturating practice-response gain), and we study when those parameters can actually be recovered from data. We prove a structural identifiability theorem for the associated inverse problem under explicit excitation conditions, with constructive closed-form recovery for the two-concept case, together with monotonicity, robustness and L-stability results. We derive a semi-implicit L-stable scheme for the dissipative subsystem and a batched solver numerically equivalent to the per-trajectory formulation (bit-exact predictions, gradients to $10^{-10}$) yet two orders of magnitude faster, making estimation feasible on cohorts of $10^5$ learners. The empirical study is two-sided. Under the theorem's excitation conditions, synthetic recovery is exact: parameters to machine precision, prerequisite structure at $F_1 = 1.0$. On large observational benchmarks it is not. An apparently strong recovery, with forgetting rates correlating with topic difficulty at Spearman $\rho = 0.83$, is refuted by four independent controls: it survives destroying the temporal order of the data, is matched by a classical Bayesian baseline, and is unaffected by removing real timestamps. We trace this to the stationary structure of the model and show that it is the degeneration the theorem predicts in the absence of designed excitation. The result delineates a sharp boundary between identifiable and unidentifiable regimes and yields a validation protocol for interpretability claims.

1. Introduction

The paper studies when mechanistically meaningful learning parameters can be recovered, finding that designed excitation permits recovery while observational data can produce misleading parameter interpretations. It frames excitation design and controls for interpretability claims as central to distinguishing recovery from degeneration.

1. Introduction

  • The paper proves exact recovery under designed excitation, but observational benchmarks lack the required practice regimes and yield parameter degeneration instead.The conditions include rest intervals and isolated, non-constant practice of each concept.Source: §1 · pp. 2–3
  • Spearman ρ = 0.83 between recovered forgetting rates and topic difficulty is reproduced by a Bayesian baseline, survives temporal shuffling, and is unaffected by real timestamps.The authors trace this apparent recovery to stationary equations that make decay absorb per-topic response frequency.Source: §1 · p. 3

2.1. Knowledge tracing: from interpretable to accurate

  • Knowledge tracing infers latent learner state from response histories, while deep models gain predictive accuracy at the cost of mechanistic state interpretation.Existing forgetting terms are often heuristic, motivating a test of whether estimated rates are actually identifiable.Source: §2.1 · p. 4

2.2. Identifiability in student modelling

  • The contribution extends identifiability analysis from discrete-time hidden Markov models to coupled continuous-time dynamics whose recoverability depends explicitly on the practice schedule.On observational benchmarks, the nominal forgetting rate can encode per-topic answer frequency.Source: §2.2 · p. 4

2.3. Structural identifiability and excitation in dynamical systems

  • The practice schedule acts as the system input, and the theorem specifies excitation regimes for unique recovery, with closed-form reconstruction for two concepts.The paper assesses standard educational logs and finds they do not provide sufficient excitation.Source: §2.3 · p. 5

2.4. Validating “interpretability”: current practice

  • A shuffle control and a simple-baseline control are proposed and applied because external-reference correlations alone cannot distinguish parameter recovery from artefact.The reported forgetting-rate correlation with topic difficulty survives temporal shuffling and is matched by a classical Bayesian baseline.Source: §2.4 · p. 5

2.5. Physics-informed and continuous-time models

  • The model hard-constrains its hypothesis class to a dissipative ODE rather than imposing the governing equation as a residual penalty.The paper reports no prior application of structural identifiability analysis to a continuous-time knowledge-dynamics model.Source: §2.5 · pp. 5–6

3. The Cognitive-PINN model

The model represents concept mastery as a continuous-time state shaped by practice, forgetting, and prerequisite transfer. Its structured transfer matrix and observation model support estimation, while stationary behavior cautions that mechanistic parameter names do not guarantee mechanistic recovery.

3.1. State and governing equation

  • Mastery evolves in a unit-bounded state under saturating practice, exponential forgetting without input, and transfer from practiced concepts to linked concepts.The stochastic term represents trial-to-trial fluctuation and is not estimated in the analysis.Source: §3.1 · p. 6

3.2. Structure of the transfer matrix

  • The transfer matrix is constrained to nonnegative coefficients, and its scaling controls spectral radius and integration stability.Experiments use an all-ones prerequisite mask rather than supplying an expert graph that would make structural validation circular.Source: §3.2 · p. 7

3.3. Observation model

  • The latent mastery state generates graded response observations through item properties, while the dynamics fix its scale and location.The theorem is structural under continuous observation; practical identifiability from sparse binary observations is assessed separately.Source: §3.3 · p. 7

3.4. Estimation problem

  • Estimation fits the response likelihood using trajectories constrained to solve the governing ODE, with zero residual weight in the experiments.An ℓ1 penalty on the transfer factor can suppress off-diagonal transfer and thus the structure a recovery experiment tests.Source: §3.4 · pp. 7–8

3.5. Interpretation and its limits

  • Although parameters are named for forgetting, transfer, and practice responsiveness, the steady-state response can pressure the forgetting rate to reproduce topic success frequency.Whether an estimate represents forgetting or reparameterizes difficulty remains an identifiability question.Source: §3.5 · p. 8

4. Structural identifiability

For the two-concept model, the theorem gives unique parameter recovery under rest and isolated, non-constant practice of each concept. Its constructive recovery also identifies practical conditioning limits and the theorem’s scope.

4.1. Setting and definition

  • The model treats mastery and practice as observed, and defines structural identifiability by whether indistinguishable parameterisations must be equal.Discrete-emission observation is addressed separately in Lemma 3.Source: §4.1 · p. 8

4.2. Excitation conditions

  • The schedule must include a positive-length rest interval and positive-length isolated, non-constant practice intervals for both concepts.Varying practice intensity separates the saturating gain from the transfer weight; a single intensity cannot distinguish weak response from weak drive.Source: §4.2 · pp. 8–9

4.3. Two lemmas

  • The unit-box lemma keeps mastery within (0, 1), while the strictly decreasing response ratio maps operating points bijectively to (1, r).The response ratio distinguishes saturation operating points that a constant practice schedule cannot identify.Source: §4.3 · p. 9

4.4. The identifiability theorem

  • Rest plus isolated, non-constant practice of each concept makes the two-concept model structurally identifiable, with closed-form recovery of every parameter.The proof gives an explicit reconstruction from observed trajectories and schedules.Source: §4.4 · pp. 9–10

4.5. Consequences

  • The closed-form formulas provide an estimator initialisation and diagnostic; reconstruction worsens with short rests, boundary mastery, or practice-intensity ratios near one.A minimal informative K = 2 design requires seven well-placed observations.Source: §4.5 · p. 10
  • The theorem concerns noise-free observations under an admissible schedule, not stochastic terms, sparse discrete observations, or misspecified Qprior.Extension to arbitrary K is conjectured; the required schedule is one an instructor could impose but learners do not follow spontaneously.Source: §4.5 · p. 11

5. Numerical methods

The forward dynamics separate practice-driven acquisition from persistent dissipation, which can make explicit integration restrictive during long gaps. A semi-implicit scheme treats dissipation implicitly and is L-stable on that subsystem.

5.1. Discretisation of the forward problem

  • Dissipation acts continuously while acquisition acts only during practice, and their rates can differ substantially across concepts.The paper uses explicit fourth-order Runge–Kutta with h = 0.25 on the interaction grid in its experiments.Source: §5.1 · p. 11
  • For long unpractised intervals, the semi-implicit scheme treats dissipation implicitly and acquisition explicitly, avoiding the explicit scheme’s restrictive step-size stability limit.The method is straightforward to evaluate because the matrix to invert is diagonal.Source: §5.1 · p. 11
  • The semi-implicit scheme is A- and L-stable for dissipation at every step size and preserves the invariant box when h∥a∥∞≤1.Its dissipative amplification factor is (1 + z)^-1 and tends to zero as z grows.Source: §5.1 · p. 11

Comparison of the schemes on stiff regimes

On a 30-day no-practice gap, the semi-implicit scheme remains stable at tested steps while explicit Runge–Kutta can leave the mastery range. The batched formulation preserves predictions and gradients while substantially reducing runtime.

Comparison of the schemes on stiff regimes

5.2. The inverse problem: adjoint sensitivity

  • Checkpointed adjoint sensitivity solves an augmented system backwards in time to reduce the memory cost of parameter updates.The paper uses direct backpropagation in its experiments because batching already removes the stated bottleneck.Source: §5.2 · p. 12

5.3. A batched reformulation with provable equivalence

  • On the same cohort and hardware, batching reduced epoch time from 6.2 hours to 16 seconds, with bit-exact predictions and objective and gradients agreeing within 8.3 × 10^-10.Transfer-parameter gradients agreed to 1.5 × 10^-10.Source: §5.3 · p. 13

5.4. Non-uniform time grids

  • Real timestamps break the shared integer time axis, so the method reparameterises and quantises node times to avoid divergent integration.Without this treatment, training produced non-finite objective values and chance-level predictions.Source: §5.4 · p. 13

6. Experimental setup

The experiments use synthetic cohorts and four public benchmarks, with common student-level folds and corrected significance tests. Parameter-recovery claims are tested against controls that disrupt temporal structure or vary the time and baseline comparisons.

6.1. Data

  • Evaluation covers synthetic cohorts and ASSISTments-2009/2015/2017 and Junyi-2015, with duplicates removed from ASSISTments-2009 and a fixed 25,000-student Junyi subsample.Models share student-level folds and identical (student, step) keys; significance uses DeLong and student-clustered bootstrap tests with Holm correction.Source: §6.1 · p. 13

6.2. Protocol for parameter validation

  • The validation protocol tests recovery with shuffled interaction order, a classical BKT baseline, real elapsed time, and a run with real time switched off.Shuffling preserves per-topic frequencies while destroying temporal structure.Source: §6.2 · pp. 13–14

7. Results

Synthetic excitation yields exact recovery, but observational data produce parameter patterns that controls show do not reflect recovered learning mechanisms. The results locate the difference in the data’s excitation and show that predictive accuracy is not the model’s contribution.

7.1. Synthetic data: the theory holds under excitation

  • For K = 2, reconstruction reaches relative error 10^-15; for K = 5, the prerequisite graph is recovered at F1 = 1.0, with AUC = 0.717 against an oracle ceiling of 0.720.These synthetic cohorts use the excitation schedule of Theorem 1.Source: §7.1 · p. 14

7.2. Predictive accuracy: a quantified trade-off

  • Cognitive-PINN is less accurate than DKT everywhere and less accurate than BKT on two of three datasets, reflecting its constraint to mechanistically meaningful parameters.The matched-AUC table reports that all pairwise differences are significant (DeLong, p ≈0).Source: §7.2 · p. 14, Table 2 · p. 14

7.3. Observational data: an apparent recovery

  • On Junyi, recovered forgetting rates correlate with topic difficulty at Spearman ρ = 0.83 ± 0.01 across five seeds, an apparent recovery signal later challenged by controls.The partial correlation is 0.78 ± 0.01 after controlling for observations per topic.Source: §7.3 · p. 15, Figure 1 · p. 15

7.4. Four controls refute the recovery interpretation

  • Across four controls, the observational forgetting-rate correlation survives temporal shuffling, is reproduced by BKT, and is unchanged by removing timestamps; structural recovery is weak and unstable.Against the expert graph, edge-AUC is 0.570 ± 0.026 over five seeds, and the sparser reference graph yields no significant signal.Source: §7.4 · pp. 15–16, Table 3 · p. 15, Figure 2 · p. 16

7.5. Why the recovery degenerates

  • The model’s stationary structure lets forgetting rates absorb per-topic response error, while genuine forgetting signal is concentrated in 2.4% of interactions with gaps of a day or more.Topic-controlled accuracy falls from 0.751 for gaps below one hour to 0.589 for gaps beyond thirty days; the remaining 97% of within-session interactions dominate the likelihood.Source: §7.5 · p. 16

8. Discussion

The discussion frames recovery as a boundary governed by observation design, not data volume, and recommends controls for interpretability claims. It also identifies limits on the theorem’s scope and proposes designed excitation as a next step.

Robustness and observational data

  • Short rest intervals, boundary mastery, and nearly equal practice intensities worsen reconstruction conditioning; observational logs exhibit short gaps and near-constant practice.The robustness bound is C†(ϵ + δ), with its constant degrading in these regimes.Source: §8 · p. 17

A sharp boundary and validation protocol

  • Exact recovery under excitation contrasts with observational recovery that degenerates into answer-frequency reparameterisation, so data scale cannot substitute for observation design.The discussion recommends a shuffle control and a simple-baseline control for interpretability claims.Source: §8 · p. 17

Limitations

  • The theorem is constructive for K = 2, while K = 5 provides empirical generalisation evidence and the general-K case remains open; external structural validation uses one dataset.The stochastic term is not estimated, and the real-time variant uses a log-compressed, quantised grid.Source: §8 · p. 17

Future work

  • A modest live-course intervention with scheduled rests and isolated, varied-intensity practice could supply the excitation conditions; the adjoint scheme also enables on-device estimation without raw-log transmission.The proposed intervention is a future direction, not a reported evaluation.Source: §8 · p. 18

9. Conclusion

The paper contrasts exact recovery under the theorem’s excitation conditions with degeneration on observational benchmarks. It identifies a testable boundary between identifiable and unidentifiable regimes and recommends controls for interpretability claims.

9. Conclusion

  • ρ = 0.83 on observational benchmarks is not successful recovery: four controls show the signal is reproduced by a classical baseline and survives temporal-order destruction.The apparent recovery is traced to a reparameterisation of per-topic answer frequencies and is unaffected by removing temporal information.Source: §9 · p. 18
  • Under the theorem’s excitation conditions, the framework recovers parameters and coupling structure exactly.The paper also presents a dissipative ODE model, identifiability theorem, supporting stability and robustness results, an L-stable scheme, and a batched solver two orders of magnitude faster.Source: §9 · p. 18
  • The results delineate a testable boundary between identifiable and unidentifiable regimes and motivate controls for claims that parameters are interpretable.The paper recommends controls capable of detecting the documented failure.Source: §9 · p. 18

CRediT authorship contribution statement

The contribution statement credits Kostanian with the main research and software work and Beklaryan with supervision and review. Code and benchmark datasets are public; raw learner logs are not redistributed.

CRediT authorship contribution statement

Data availability

Appendix A. Completion of the proof of Theorem 1

The proof completes two-concept recovery using isolated practice and specified observations. It also bounds the observed ratio and identifies the informative instants required by the design.

Appendix A.1. Step 4

  • On I2, where p1 is zero, two distinct p2 practice levels recover α2 and A22 from the second-component dynamics.Source: Appendix A.1 · p. 19

Appendix A.2. Range of the observed ratio

  • The observed ratio ρ lies strictly between 1 and r; the excluded limiting regimes correspond to vanishing gain and complete saturation.Source: Appendix A.2 · p. 19

Appendix A.3. Minimal informative design

  • The noise-free K = 2 design uses seven informative instants: two on rest intervals, two distinct practice intensities per concept, and one additional instant per concept.Source: Appendix A.3 · p. 19

Appendix B.1. Setting

  • The robustness setting assumes state and derivative errors bounded by ϵ and δ, respectively, with mastery and distance from the upper boundary both bounded away from zero.Source: Appendix B.1 · p. 19

Appendix B.3. The observable g1

  • The observable g1 is written as (k̇1 + Λ1k1)/(1 − k1), separating its numerator from the remaining mastery gap.Source: Appendix B.3 · p. 19

Appendix B.4. The gain

The appendix quantifies reconstruction conditioning and noise sensitivity, gives practical excitation guidance, and states the grid conditions behind batched equivalence.

Appendix B.4. The gain

  • The observed gain-ratio reconstruction error is bounded linearly by ϵ + δ, and the resulting diagonal and off-diagonal parameter estimates also have linear bounds.Source: Appendix B.4 · p. 20

Appendix B.5. What the constant contains

  • Reconstruction degrades with short rests, mastery near either boundary, and practice intensities close together; intensity variation therefore controls conditioning, not just uniqueness.Source: Appendix B.5 · p. 20

Appendix B.6. Empirical range of validity

  • For ϵ ≤ 0.01, reconstruction-error growth is compatible with the linear bound and noise-free reconstruction is exact; beyond ϵ ≈ 0.02, error rises abruptly.The abrupt increase occurs when the perturbed ratio leaves the inversion’s valid range, so the inversion has no solution.Source: Table B.4 · p. 20, Appendix B.6 · p. 20

Appendix B.7. Design corollary

  • The design corollary calls for non-negligible rest intervals, avoiding mastery floors or ceilings, and varying practice intensity by at least a factor of two.Source: Appendix B.7 · p. 20

Appendix C. Proof of Proposition 2

  • On integer interaction grids, batched and per-trajectory paths compute the same objective and gradients; continuous grids can make the union-grid computation memory-infeasible and unstable.Source: Appendix C · p. 21
Loading 2610.09889v1…