PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEAN: Psychometrically Consistent Incremental Cognitive Diagnosis under Concept-Space Expansion via Architectural Isolation

Tao He, Jinxing Xiang, Fan Jiang

arXiv:2610.02278v1cs.LG

TL;DR

Incremental cognitive diagnosis must accommodate newly introduced concepts while preserving historical diagnoses for the same inputs. CLEAN uses Q-matrix-guided architectural isolation, and experiments report zero representation drift with anchor-level old-item metrics across three datasets.

  • Problem

    New items can introduce unseen concepts, but existing incremental CD methods assume a fixed concept space and soft constraints do not guarantee unchanged historical diagnoses.

    Source: §1 · p. 1

  • Method

    CLEAN uses strict topological bipartition, freezes historical diagnostic functions, masks shared columns, and adds expandable branches to isolate historical pathways.

    Source: §1 · p. 2, §4.2 · pp. 4–5

  • Results

    Across Junyi, ASSISTments, and Math1, CLEAN achieves RD = 0 and anchor-level old-item metrics while remaining competitive with or superior to continual-learning baselines on new items.

    Source: §1 · p. 2, §5.2 · p. 8, §6 · p. 8

  • Takeaways & Limitations

    Architectural isolation preserves historical diagnostic outputs pointwise without replaying old responses while retaining strong plasticity for new concepts.

    Source: §6 · p. 8

    Multi-stage streams, long-horizon consistency, cumulative capacity growth, and repeated item and concept expansion remain for future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Cognitive diagnosis (CD) is a fundamental task in intelligent education that profiles learner proficiency over knowledge concepts. In real-world learning platforms, newly added items continually introduce previously unseen concepts, necessitating dynamic expansion of the underlying concept space. Yet existing incremental CD models assume a fixed concept space, allowing gradients from new items to overwrite historical pathways and induce catastrophic forgetting. More critically, these methods rely solely on soft constraints to preserve historical diagnoses. Such constraints may fail to satisfy the requirement of diagnostic invariance after incremental updates, a requirement known as psychometric consistency in cognitive diagnosis. Therefore, we propose CLEAN (Continual Learning with Expandable and Architecturally Isolated Networks), a novel incremental CD framework supporting concept-space expansion while providing structural guarantees for pointwise invariance of historical diagnoses. Specifically, CLEAN first introduces a strict topological bipartition protocol, freezes historical diagnostic functions and applies deterministic orthogonal column masking to sever gradient interference. Second, to accommodate concept expansion, expandable full-rank branches with micro-variance initialization are deployed to learn novel concepts. Finally, to verify that this architectural design achieves invariance by construction, we formalize Representation Drift (RD) to quantify the perturbation of historical traits. Extensive experiments on three large-scale educational datasets demonstrate that CLEAN achieves zero RD, preserving old-item metrics identically to static anchors through architectural isolation while remaining competitive with or superior to strong continual-learning baselines on new items.

1 Introduction

Existing incremental cognitive diagnosis methods do not accommodate concept-space expansion while guaranteeing that historical diagnoses remain unchanged. CLEAN uses the Q-matrix to isolate historical pathways structurally and reports zero representation drift with anchor-level retention.

1 Introduction

  • New items can introduce unseen concepts, requiring column-wise Q-matrix expansion that existing incremental CD work does not support.Fine-tuning a widened trait layer risks overwriting pathways that generate historical diagnoses.Source: §1 · p. 1
  • Aggregate old-task accuracy and soft stability constraints do not guarantee unchanged diagnoses for the same historical response input.Replay also requires storing historical responses, which privacy and storage constraints may preclude.Source: §1 · p. 1
  • CLEAN uses Strict Topological Bipartition to freeze historical GDFs, mask shared columns, and isolate expandable branches so invariance is a computational-graph property.The Q-matrix topology guides the isolation design.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • CLEAN attains RD = 0 and anchor-level old-item metrics across Junyi, ASSISTments, and Math1, while no baseline surpasses it on new items without replay or substantial representation drift.The three datasets span dense to highly sparse responses.Source: §1 · p. 2

2 RELATED WORK

Prior cognitive diagnosis and continual-learning approaches leave a gap between concept-level expansion and pointwise preservation of historical diagnoses. CLEAN addresses both by making historical invariance structural rather than relying on soft stability objectives.

Cognitive Diagnosis Models

  • Transductive and generative cognitive diagnosis models both assume a fixed concept space, despite generative functions supporting inference for unseen learners [32].Transductive models require global retraining when learners or items arrive because they use entity-specific parameters.Source: §2 · p. 2

Incremental Cognitive Diagnosis

  • Existing incremental CD methods expand entities rather than concepts, and preserve historical diagnoses approximately through soft stability terms [34].ICD explicitly assumes an unchanged concept set.Source: §2 · p. 2

Continual Learning and the Measurement of Forgetting

  • Regularization and replay suppress but do not eliminate drift, while task-identity-based architectural isolation misses Q-matrix concept topology [15] [19] [1] [2] [4] [22] [26] [28] [12] [13] [20] [35] [37].Aggregate accuracy can remain stable even as individual learners’ historical proficiency changes.Source: §2 · p. 3

3 PRELIMINARY

The section introduces a generative cognitive-diagnosis backbone that derives concept-aligned traits from response logs and predicts responses through Q-gated aggregation. It then defines incremental expansion and Representation Drift (RD) to quantify changes in historical traits, while noting that trait stability alone does not ensure unchanged old-item predictions.

3.1 Generative Cognitive Diagnosis Backbone

  • The Generative Diagnosis Function derives learner and item traits from response logs, which are Q-gated and projected before an item response function predicts correctness.The learner-side projection is constrained to be non-negative under the monotonicity assumption; training uses binary cross-entropy on observed interactions.Source: §3.1 · p. 3

3.2 Incremental Cognitive Diagnosis Setting

  • Incremental learning expands both the item and concept dimensions of the Q-matrix, requiring new-item predictions and proficiency estimates while maintaining stability of historical traits.The initial anchor model is trained on historical data in the old item–concept space, and deviations from the zero-drift condition are quantified as RD.Source: §3.2 · p. 3

3.3 Quantitative Metric: Representation Drift

  • RD measures the shift in historical learners’ traits within the old-concept subspace; RD = 0 indicates pointwise invariance, but unchanged old-item predictions also require Orthogonal Column Masking and Shared Bias Freezing.The metric’s normalization by √K_old reduces dimensionality effects and enables comparison across models.Source: §3.3 · p. 3

4 Methodology

CLEAN combines STB-based topological reordering, expandable diagnostic functions, and aggregation-layer isolation to preserve historical traits while learning newly introduced concepts. Its design separates old and new diagnostic pathways and freezes historical aggregation parameters.

4.1 Overview of CLEAN

  • CLEAN combines STB-based reordering, expandable GDFs, and an isolated aggregation layer to preserve historical traits while incremental branches learn new concepts.The overview states that historical outputs remain unchanged and historical traits have RD = 0.Source: §4.1 · p. 4

4.2 STB Protocol

  • STB reorders Q-matrix item rows and concept columns so historical items have no links to new concepts, while new items may assess both old and new concepts.This topology is based on expert-annotated Q-matrices and provides the basis for later orthogonal column masking.Source: §4.2 · pp. 4–5, Figure 3 · p. 4

4.3 CLEAN-GDF

  • New GDF branch weights use micro-variance initialization, starting near the frozen reference parameters while retaining gradients to learn new concepts.The new weights use a truncated Gaussian with variance 10^-3 times that of the frozen reference parameters; LoRA factors use small nonzero weights to avoid zero-gradient death.Source: §4.3 · p. 5
  • Learner and item traits use separate expansion pathways; only learner inputs are partitioned into old- and new-item blocks to protect historical ability dimensions.The learner pathway concatenates frozen and new outputs, while the item pathway processes the complete learner-wise response column in both branches.Source: §4.3 · p. 5

4.4 CLEAN Aggregation Layer

  • The aggregation layer projects concatenated old and new concept traits into a fixed-dimensional representation, while preserving the historical aggregation subspace at its anchor state.It uses deterministic orthogonal column masking for concept-wise weights and freezes the shared bias.Source: §4.4 · p. 5

4.4.1 Deterministic Orthogonal Column Masking.

  • For historical items, STB zeros new-concept contributions, while orthogonal column masking prevents historical aggregation-weight columns from drifting during incremental training.The incremental branch uses responses to new items rather than hidden features from the historical subnet, keeping their gradient graphs disjoint.Source: §4.4.1 · pp. 5–6

4.4.2 Parameter Bias Freezing.

  • The shared aggregation bias is excluded from the optimizer and frozen at its anchor value, complementing weight masking to preserve bitwise-identical historical outputs.The paper states that both mechanisms are indispensable and that removing either degrades old-item prediction performance.Source: §4.4.2 · p. 6

5 EXPERIMENTS

The experiments evaluate CLEAN under item- and concept-space expansion across three datasets, comparing retention on old items with adaptation to new items. CLEAN preserves anchor-level old-item accuracy with zero representation drift, while ablations show that zero drift alone does not ensure unchanged predictions.

5.1 Experimental Setup

  • Junyi, ASSIST, and Math1 span 71.36%, 99.64%, and 0.00% response sparsity, respectively, covering normal, extremely data-scarce, and interaction-dense settings.The datasets use 712 × 39, 17,746 × 123, and 20 × 11 Q-matrices, respectively.Source: §5.1 · p. 6
  • The two-stage protocol trains on old items first, then expands to new items and evaluates separate old-item and new-item test sets.Random-Split places the same learners in train, validation, and test sets, isolating expansion from cold-start learner adaptation; metrics include ACC, F1, RMSE, ACCall, and RD.Source: §5.1 · p. 6, §5.2 · p. 7
  • Same-backbone comparisons use Full-Replay Oracle, X-DER, C-LoRA, and CLEAN; EWC, DER++, and ICD serve as cross-backbone references.The cross-backbone methods retain their original implementations because adapting them to expandable G-NCDM requires structural modifications beyond the original methods.Source: §5.1 · p. 6

5.2 Experimental Results

  • Across three datasets, CLEAN-Full and CLEAN-LoRA match anchor ACCold with RD = 0; CLEAN-Full ranks second only to Full-Replay Oracle on Junyi and a0910.On Math1, CLEAN-Full records ACCnew of 72.25 while retaining anchor-level ACCold of 72.14.Source: Table 1 · p. 7, §5.2 · pp. 7–8
  • Only CLEAN-Full shows no observable historical-representation displacement on Math1; the other visualized methods shift after incremental training.The UMAP comparisons show four methods on the same G-NCDM backbone, with points colored by old-item accuracy; the finding is consistent with CLEAN-Full's RD = 0.00.Source: Figure 5 · p. 7, §5.2 · p. 8

5.3 Ablation study

  • Although every ablation has RD = 0, ACCold falls from 69.42 to 60.22 without OCM, 55.99 without SBF, and 44.46 without both.The results show that freezing the historical GDF alone preserves the measured trait subspace but not necessarily old-item predictions; new-item metrics vary only moderately.Source: Table 2 · p. 8, §5.3 · p. 8

6 CONCLUSION

CLEAN encodes psychometric consistency as a hard architectural constraint and combines isolation with expandable branches for new concepts. Experiments report zero representation drift and anchor-level old-item performance; extending the evaluation to multi-stage streams and long-horizon consistency remains future work.

6 CONCLUSION

  • Across Math1, Junyi, and ASSISTments, CLEAN achieves RD = 0, maintains anchor-level old-item performance, and remains competitive with or superior to continual-learning baselines.The conclusion describes these datasets as representing dense, moderately sparse, and highly sparse response settings.Source: §6 · p. 8
  • Future work will extend CLEAN to multi-stage continual streams and investigate long-horizon consistency, cumulative capacity growth, and repeated item- and concept-expansion adaptation.Source: §6 · p. 9

A.1 User-Split Results

On ASSISTments 2009–2010 under User-Split, CLEAN-LoRA preserves historical performance exactly while achieving the best new-item metrics. Non-CLEAN baselines show nonzero representation drift.

A.1 User-Split Results

  • C-LoRA-GNCDM generally performs worse than original C-LoRA on old and new items, although their RMSEnew values are nearly identical.Replacing the original backbone with G-NCDM alone does not improve predictive performance.Source: §A.1 · p. 9
  • The User-Split protocol uses disjoint learner groups across training, validation, and test sets to assess diagnosis of unseen learners from limited support responses.Methods process support information differently but are evaluated on the same held-out query interactions.Source: §A.1 · pp. 9–10
  • CLEAN-LoRA achieves ACCnew = 69.92, F1new = 78.58, and RMSEnew = 44.76 with RD = 0.00, while exactly matching anchor-level old-item metrics.ACCold = 70.24, F1old = 79.65, and RMSEold = 45.22.Source: Table 3 · p. 10, §A.1 · p. 10
  • Full-Replay Oracle reaches ACCnew of 67.82, 2.10 percentage points below CLEAN-LoRA, while its RD is 3.44.Other listed non-CLEAN baselines also exhibit nonzero drift and generally weaker new-item performance.Source: §A.1 · p. 10

A.2 Detailed Metric Results on Each Dataset

Across Junyi-Random, ASSISTments a0910, and Math1, CLEAN preserves anchor-level historical metrics with zero representation drift while retaining competitive new-item adaptation.

A.2 Detailed Metric Results on Each Dataset

  • Across all three datasets, CLEAN-Full and CLEAN-LoRA exactly match the G-NCDM anchor on ACCold, F1old, and RMSEold, consistently achieving RD = 0.This preserves historical trait representations and old-item prediction performance after incremental updates.Source: §A.2 · p. 10
  • On Junyi-Random, CLEAN-Full records ACCnew of 74.53, F1new of 81.10, and RMSEnew of 41.83, remaining close to Full-Replay Oracle.On ASSISTments a0910, its corresponding metrics are 72.21, 79.61, and 43.45.Source: §A.2 · p. 10
  • On Math1, replay-based methods achieve stronger new-item performance, while CLEAN-Full retains anchor-level old-item metrics with zero representation drift.Across datasets, CLEAN maintains competitive adaptability while providing strict historical stability.Source: §A.2 · p. 10

A.3 Sensitivity Analysis of the Residual Mixing Coefficient 𝛼

The sensitivity analysis finds dataset-specific accuracy optima at relatively small values of α. Larger values can limit adaptation to incremental data.

A.3 Sensitivity Analysis of the Residual Mixing Coefficient 𝛼

  • The coefficient α balances the implicitly learned representation against the explicitly extracted Q-matrix representation.Smaller α places greater weight on θimp, while larger α emphasizes θexp.Source: §A.3 · p. 10
  • The selected values are α = 0.2 for Math1 and α = 0.1 for Junyi and ASSIST09-10; excessive reliance on foundation-stage information limits incremental-data adaptation.Relatively small values provide a better trade-off between retaining learned information and incorporating new information.Source: §A.3 · p. 10
  • Accuracy peaks at α = 0.2 on Math1 and α = 0.1 on Junyi and ASSIST09-10; accuracy declines beyond these optima.The figure presents sensitivity analysis of α under Random-Split.Source: §A.3 · p. 10, Figure 7 · p. 11

Related papers

Loading 2610.02278v1…