Beyond ID Embeddings: Process-Grounded Language Modeling for Cognitive Diagnosis

1 Introduction

• Traditional CDMs represent students, exercises, and concepts with arbitrary IDs, limiting access to content semantics and use of new exercise text without sufficient response records.This ID-centric approach primarily fits behavioral regularities rather than capturing underlying cognitive-state structure.Source: §1 · p. 1
• Language-enhanced CDMs often retain IDs as primary anchors and compress learner histories into static profiles that may obscure target-relevant evidence and dynamic mastery.These constraints persist even as prior work incorporates knowledge graphs, LLMs, or text encoders.Source: §1 · p. 2
• PLCD uses LLM-generated concept schemas and cognitive process graphs as priors, retrieves target-relevant historical responses, and maps semantic evidence into diagnostic representations.Its process-level contrastive learning distinguishes mastered from unmastered concepts, while a psychometric outcome head accounts for guessing and slipping.Source: §1 · p. 2
• Across three real-world educational datasets, PLCD significantly outperforms ID-based models in learning-outcome prediction.Qualitative analyses also distinguish cognitive deficits such as semantic translation errors and procedural failures.Source: §1 · p. 2

2.1 Structured Cognitive Evidence Construction

• LLMs convert concept and exercise content into schemas and cognitive process graphs that represent concepts, operations, solution steps, and potential misconceptions.Exercise graphs go beyond concept membership to capture required reasoning operations and error-prone steps.Source: §2.1 · p. 2
• For each target exercise, PLCD retrieves earlier response records using graph similarity and Q-matrix concept overlap, retaining each attempt’s graph, correctness, and elapsed time.The retrieved evidence represents performance on prior exercises requiring similar cognitive operations and can indicate implicated process-level errors.Source: §2.1 · p. 3

2.2 Language-to-Cognition Mapping for Diagnosis

• The mapper binds feed-forward experts to predefined cognitive process types and gates them using both the input representation and the target exercise’s process prior.A top-k gate activates the most relevant process experts, keeping their contributions traceable to explicit reasoning operations.Source: §2.2 · pp. 3–4
• Process-level contrastive learning uses retrieved historical success and error evidence to weight positive and negative evidence for the processes required by the target exercise.Incorrect responses supply negative evidence only for required processes on which the student has historically been weak, rather than treating every process in an exercise as failed.Source: §2.2 · p. 4

2.3 Inferring Learning Outcomes from Cognitive States

• PLCD derives concept-specific student knowledge states and exercise difficulty factors to estimate correctness, then adjusts the diagnosis-based score for exercise-level guessing and slipping.The objective keeps supervised response calibration explicit while preserving the language-derived cognitive prior.Source: §2.3 · p. 5

3.1 Experimental Setup

• The evaluation uses Junyi, XES3G5M, and MOOC, reporting ACC, AUC, and RMSE for performance prediction and Brier Score and ECE for component-level calibration.The paper compares representative ID-based and knowledge-enhanced diagnosis models.Source: §3.1 · p. 5

3.2 Performance Comparison and Ablation Study

• PLCD consistently outperforms state-of-the-art baselines, including knowledge-enhanced CDMs, on student performance prediction.Table 1 reports experimental results and ablations; its caption specifies that the best results are highlighted and significance is tested across ten runs.Source: §3.2 · p. 5, Table 1 · p. 6
• Removing structured evidence, target-conditioned memory, the process-grounded mapper, or process-level contrastive learning degrades performance, with target-conditioned memory especially helpful on XES3G5M and MOOC.The ablations indicate that raw text alone is insufficient, static response aggregation blurs target-relevant evidence, language embeddings need cognitive alignment, and process supervision improves state discriminability.Source: §3.2 · pp. 5–6

3.3 Cold-start and Sparsity Analysis

• On XES3G5M, PLCD exceeds RCD and KCD at every tested sparsity level, with a larger advantage as responses become sparser.In cold-start settings, it exceeds KCD by 4.60 ACC points for new concepts, 4.00 ACC points for new exercises, and 4.52 ACC points when the Q-matrix is missing; it remains below Oracle.Source: §3.3 · pp. 6–7, Figure 3 · p. 6, Table 2 · p. 6

3.4 Mechanism Verification

• Semantic-plus-Q retrieval has the highest Q-Jac.@5, while its Res.Corr.@5 is much higher than the label-shuffled variant, supporting the value of retrieved correctness patterns.Semantic-only and concept-overlap retrieval perform below PLCD in ACC; random retrieval performs worst among retrieval variants.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 Language-to-Cognition Mapping Analysis

• Process-prior conditioning improves gate-prior alignment and expert entropy, while process-level contrastive learning further improves prediction and grounding metrics.The analysis compares DA-MoE routing variants with and without process-prior conditioning and contrastive learning.Source: §3.5 · p. 7, Table 4 · p. 7

3.6 Cognitive Scaffolding in Latent Space

• The mapped exercise representations form a continuous manifold, with domain clusters, transitions between skills, and cross-domain proximity for shared cognitive operations.The exploratory t-SNE analysis includes geometry and fraction clusters, arithmetic-to-fraction transitions, and examples of shared operations such as spatial reasoning.Source: Figure 4 · p. 7, §3.6 · p. 8

3.7 Diagnostic Report Analysis

• For a held-out student, PLCD produces sharper diagnostic estimates than ID-based models, whose predictions remain mainly in the 0.4–0.6 range.The report attributes this contrast to structured concept and process priors combined with target-conditioned retrieval of responses on cognitively relevant exercises.Source: §3.7 · p. 8, Figure 5 · p. 8

• PLCD avoids learned student and exercise ID embeddings by using structured language-derived cognitive priors.The framework pairs those priors with explicit response calibration.Source: §4 · p. 8
• Experiments show improved diagnostic accuracy and robustness in sparse and cold-start settings when response records calibrate the priors.Source: §4 · p. 8
• Qualitative analyses show that PLCD captures psychologically meaningful latent structure and distinguishes semantic translation errors from procedural failures.Source: §4 · p. 8

• PLCD requires available response records to calibrate learner-specific states.Source: Limitations · p. 9
• The quality of structured cognitive evidence may depend on the clarity of exercise text and concept descriptions.Source: Limitations · p. 9
• Because true mastery is unobservable in real-world datasets, evaluation relies mainly on response prediction and related calibration metrics.Source: Limitations · p. 9
• The findings do not establish causal learning mechanisms or make PLCD a complete substitute for longitudinal educational assessment.Source: Limitations · p. 9

Ethical considerations

• The study uses public, anonymized datasets without collecting private student information, and positions PLCD as decision support with human oversight rather than automated high-stakes decision-making.The teacher-validation study involved paid participants; further details are in Appendix H.Source: Ethical considerations · p. 9

A.1 ID-based Cognitive Diagnosis Models

• ID-based models depend on response logs and offer limited access to content semantics, which constrains generalization to new exercises and sparse-response settings.PLCD addresses this gap with language-derived cognitive priors while retaining response records for personalized calibration.Source: §A.1 · pp. 11–12

A.2 Language- and Knowledge-enhanced Cognitive Diagnosis

• PLCD organizes diagnosis around LLM-derived concept schemas and process graphs, retrieving target-relevant response records instead of compressing learner history into a fixed ID-bound profile.A process-grounded mapper aligns this evidence with diagnostic states.Source: §A.2 · p. 12

B Process Expert Inventory

• The same process-type expert inventory is used across Junyi, XES3G5M, and MOOC, while each exercise’s process prior shapes its activated expert mixture.Source: B Process Expert Inventory · p. 12, Table 5 · p. 12

C.1 Datasets

• Evaluation uses Junyi, XES3G5M, and MOOC response records with exercise-concept correlations; the processed subsets retain valid interactions with usable exercise text and annotations.Source: §C.1 · p. 13
• The datasets focus on mathematics, whose hierarchical concepts, objective correctness labels, and explicit exercise-concept mappings support process-level evaluation and comparison.The three platforms also differ in response distributions and concept structures.Source: §C.1 · p. 13

C.2 Baselines and Evaluation Metrics

• Baselines span traditional IRT and MIRT, neural NCD, RCD, and ACD, and knowledge-enhanced KaNCD, TechCD, KCD, and DMC-CDM.The comparison accounts for reproducibility.Source: §C.2 · p. 13
• Because true mastery is unobservable, model effectiveness is evaluated indirectly through exercise-performance prediction using ACC, AUC, and RMSE.Source: §C.2 · p. 13
• Brier Score and Expected Calibration Error are tracked in component-level calibration analysis because the guess-slip head outputs calibrated response probabilities.Source: §C.2 · p. 13

C.3 Implementation Details

• The fixed 8:1:1 response-record split follows a cognitive-diagnosis protocol, with only earlier training responses eligible as memory for a target and no sequential test-time updates.Validation selects hyperparameters; test data are reserved for final reporting.Source: §C.3 · pp. 13–14
• Shared hyperparameters are tuned on validation data, with eight DA-MoE experts, TopK k = 2, and contrastive temperature τ = 0.1.The best validation AUC configuration is used for final testing.Source: §C.3 · p. 14

C.4 Statistical Uncertainty

• Main prediction results are averaged over ten independent runs; significance marks use a two-sided paired t-test comparing PLCD with the corresponding strongest non-PLCD baseline across the same ten runs, with p < 0.05.The mean and sample standard deviation are reported for PLCD.Source: §C.4 · p. 14

D Cold-start Evaluation Settings

• Cold-start tests hold out concepts, exercises, or expert Q-matrix links; held-out records and masked links are excluded from training, retrieval, and inference.New-concept targets lack learner-response calibration, while missing-Q-matrix evaluation removes explicit concept-overlap information; Oracle is trained with full data and annotations.Source: D Cold-start Evaluation Settings · pp. 14–15
• For new exercises, PLCD uses exercise text, difficulty, concept annotations, and process graphs, whereas ID-based baselines lack learned response patterns for held-out items.The Q-matrix remains available in this setting.Source: D Cold-start Evaluation Settings · pp. 14–15
• RCD is excluded from missing-Q-matrix evaluation because its relation maps require student–exercise–concept associations.Baselines requiring a Q-matrix receive the masked matrix.Source: D Cold-start Evaluation Settings · p. 15

E Memory Mechanism Verification Setup

• The memory verification setup changes only retrieval weights used to construct learner memory, leaving the other modules unchanged.Source: E Memory Mechanism Verification Setup · p. 15

E.1 Memory Strategy Variants

• Memory variants compare target-independent aggregation, random retrieval, semantic-only and concept-overlap-only retrieval, and the full Semantic+Q strategy.Label shuffling preserves retrieved exercises and each student's marginal correctness distribution while breaking exercise–response alignment.Source: §E.1 · pp. 15–16

E.2 Retrieval-level Diagnostics.

• Q-Jaccard@K assesses concept overlap in retrieved histories, while Residual Response Correlation@K tests predictive response evidence beyond a student's overall correctness level.Shuffling response labels should preserve Q-Jaccard@K but reduce residual response correlation.Source: §E.2 · p. 16

F Cognitive Grounding Evaluation Setup

• Cognitive-grounding evaluation varies whether DA-MoE uses the LLM-derived process prior and whether process-level supervised contrastive learning is applied.Source: F Cognitive Grounding Evaluation Setup · p. 16

F.1 Compared Variants

• Variants progress from ungrounded experts to process-labeled experts without prior-guided routing, prior-guided routing, and full PLCD with process-level contrastive learning.The full variant aligns learner evidence with process-specific exercise demands.Source: §F.1 · pp. 16–17

F.2 Evaluation Metrics

• Gate-Prior Cosine measures similarity between the DA-MoE gate distribution and the LLM-derived process prior; higher values indicate closer alignment.Source: §F.2 · p. 17
• KL(πe∥ge) measures divergence from the process prior to the gate distribution, with lower values indicating closer prior-following.Ungrounded MoE has no stable process semantics, so its cosine and KL values are not reported.Source: §F.2 · p. 17
• Expert Entropy measures gate-distribution entropy; higher values indicate broader expert use and help diagnose routing collapse.Source: §F.2 · p. 17

G Calibration Analysis

• Adding guess-slip parameters improves performance across all three datasets, with larger gains in Brier Score, ECE, and NLL than in ACC.Table 8 compares w/o Guess-Slip, Global Guess-Slip, and Exercise-level Guess-Slip; the reported pattern indicates stronger probability calibration than thresholded accuracy gains.Source: G Calibration Analysis · pp. 17–18
• Exercise-level Guess-Slip consistently outperforms the global correction, supporting exercise-specific rather than uniform dataset-level guessing and slipping effects.The authors attribute differences to exercise difficulty, format, distractor quality, and cognitive demand.Source: G Calibration Analysis · p. 18

H Human Teacher Validation of LLM-Generated Cognitive Process Graphs

• Five teachers independently rated 120 stratified exercises across three datasets without seeing response records, PLCD predictions, baseline outputs, or model performance.Source: H Human Teacher Validation of… · p. 18
• Teachers scored five graph dimensions on 1–5 scales; agreement analyses included Top-1, Top-2, weighted process-prior cosine, and Fleiss’ kappa.Source: H Human Teacher Validation of… · p. 18
• Gemini 3 Pro achieved the highest overall teacher score and strongest process-label agreement among generators, so it became the default; Table 9 compares generators.Source: H Human Teacher Validation of… · pp. 18–19, Table 9 · p. 19
• Teachers generally rated Gemini-generated graphs as plausible, especially for concept relevance and operation correctness; lower misconception plausibility may require teacher correction before high-stakes deployment.Source: H Human Teacher Validation of… · p. 19

I LLM Prompt Templates for Structured Cognitive Evidence

• Appendix I provides reproducible prompt templates for constructing cognitive priors before response-based calibration, but not full API logs.Source: I LLM Prompt Templates for Structured… · p. 19

I.1 Concept Schema Prompt

• The concept-schema prompt takes the target concept name as its input.Source: §I.1 · p. 19, Figure 6 · p. 20

I.2 Exercise Cognitive Process Graph Prompt

• The exercise-graph prompt takes the exercise text, its Q-matrix concept set, and difficulty level as inputs.Source: §I.2 · p. 19, Figure 7 · p. 20