PaperlayerView exact revision on arXiv

Source-linked AI summary

Prediction Is Not Calibration: Item Parameter Recovery and Representation Coupling in Knowledge Tracing

Wenrui Yuan, Stéphanie M. van den Berg, Johannes Steinrücke

arXiv:2610.30211v1cs.LGcs.AI

TL;DR

Sequential KT studies have not established whether next-response training recovers the range of item parameters represented by IRT decoders. This paper analyzes the identification mechanisms and evaluates separated sequence and parameter representations. Accurate prediction and location recovery can coexist with weak slope recovery, while a wide parameter key with a thin sequence representation improves recovery and cross-order agreement in the reported comparisons.

  • Problem

    Prior sequential KT–IRT work estimates only 1PL difficulty or evaluates richer decoders by prediction, leaving recovery of discrimination, step thresholds, and option slopes unresolved.

    Source: §2.3 · p. 6

  • Method

    The paper derives gradient and affine-invariance analyses and tests shared versus separated item representations in controlled simulations and TIMSS and EdNet benchmarks.

    Source: §1 · p. 3, §5 · p. 26

  • Results

    Accurate next-response prediction and location estimates often coexist with poor discrimination and option-slope recovery; SK(64, 8) reduces fixed-order distortions and improves cross-order rank agreement over SH(64) and SK(64, 64).

    Source: §1 · p. 3, §5 · p. 26

  • Takeaways & Limitations

    Test accuracy and model size do not determine item-parameter quality, so IRT decoder parameters require evaluation separately from response prediction.

    Source: §1 · p. 4

    Exact recovery is verifiable only in simulations from a stationary unidimensional IRT model matching the decoder family; real-data MML references are calibrated on the same response data rather than independent ground truth.

    Source: §5.4 · p. 29

Abstract

from arXiv · show

Knowledge tracing (KT) models can estimate item parameters through an item response decoder trained solely for next-response prediction, yet predictive accuracy does not guarantee parameter recovery. We study 2PL, GPCM, and NRM decoders paired with LSTM, Transformer, and DKVMN encoders. Response-model derivatives show why discrimination and option slope updates depend directly on the scale and spread of estimated ability. Affine logit identities show how item-dependent shifts and rescaling in estimated ability are absorbed into item parameters without changing response probabilities. In simulations, difficulty, mean threshold, and correct-option intercept recovery remain high under shuffled order across representation widths, whereas discrimination and keyed option slope contrast recovery require wider item representations and degrade further under fixed item order. Separating the sequence item representation (width $d_S$) from the item parameter key (width $d_M$), and pairing a wide key with a thin sequence representation ($\mathrm{SK}(64,8)$), reduces fixed-order ability distortions and improves recovery and cross-order consistency without sacrificing test accuracy or increasing model size relative to a shared width-64 baseline ($\mathrm{SH}(64)$). On TIMSS and EdNet, widening $d_M$ consistently improves agreement with an offline IRT reference, while the trade-off from narrowing $d_S$ depends on the encoder's reliance on sequence width for prediction. These findings show why item parameters, especially discrimination and option slopes, must be evaluated separately from response prediction.

1. INTRODUCTION

Neural KT models can place IRT equations at the output and train them for next-response prediction, but this does not establish that learned item parameters are calibrated. The paper analyzes why location and slope recovery differ and tests whether separating sequence representations from parameter keys improves recovery.

1. INTRODUCTION

  • Location updates depend on prediction residuals, while discrimination and option-slope updates also depend on estimated ability or ability-to-location distance.Thus, slope estimation is directly sensitive to the spread and resolution of item and ability representations.Source: §1 · p. 2
  • Item-dependent ability shifts and positive rescaling can be absorbed into item parameters along affine logit directions without changing predicted response probabilities.Under structured item order, history can encode item-specific offsets and gains in ability estimates; shared item embeddings also receive gradients through both the sequence and parameter-readout paths.Source: §1 · p. 2
  • The study evaluates three encoders and three response decoders while SK separates the parameter-key width dM from the sequence-item width dS without changing the backbone, response equation, or training objective.SH uses one item table for both paths; SK assigns separate tables to the parameter readout and sequence encoder.Source: §1 · p. 3, Figure 1 · p. 3
  • Fixed-order ability offsets and gains degrade uncorrected item-parameter recovery, while SK(64, 8) reduces these distortions and improves cross-order rank agreement over SH(64) and SK(64, 64).The study combines controlled simulations with TIMSS 2019 and EdNet-KT1 benchmarks against MML reference estimates.Source: §1 · p. 3
  • Across the width grid, widening dM from 8 to 64 improves option-slope and discrimination estimation, whereas widening dS at dM = 64 degrades simulated recovery.The dM improvement holds across all nine simulated pairs and all nine dS = 8 real-data pairs.Source: §1 · p. 3
  • The paper derives gradient asymmetries, gradient-routing equations, and affine logit invariances for 2PL, GPCM, and NRM decoders.These analyses explain why location recovery exceeds slope recovery and how narrowing dS while widening dM reduces item-order dependence.Source: §1 · p. 4

2. RELATED WORK

Prior work adds interpretability or IRT structure to knowledge tracing, but often does not identify formal item parameters or evaluate their recovery. This paper situates its question at the intersection of neural IRT estimation and sequential KT, where shared representations couple state estimation and item calibration.

2.1. INTERPRETABILITY IN KNOWLEDGE TRACING

  • Latent-state labels, prerequisite edges, and scalar difficulty modifiers are not equivalent to parameters identified through a formal IRT measurement equation.Existing KT work also uses structured representations, graph propagation, and option-level prediction.Source: §2.1 · p. 4

2.2. NEURAL ESTIMATION OF ITEM RESPONSE MODELS

  • Neural IRT methods use amortized variational inference with the IRT likelihood, while unamortized inference can recover item parameters better in small samples yet predict missing responses worse.This provides evidence that predictive accuracy and parameter recovery need not rank estimators identically.Source: §2.2 · p. 5
  • Sequential KT couples stepwise state updates to later-response gradients, and shared item representations can link history encoding with item-parameter readout.This coupling differs structurally from static variational autoencoders.Source: §2.2 · p. 5

2.3. ITEM RESPONSE MODELS AS KNOWLEDGE TRACING DECODERS

  • Prior KT–IRT decoders either estimate only 1PL difficulty or evaluate richer IRT decoders through prediction rather than item-parameter recovery.Deep-IRT fixes discrimination at 3.0; Tsutsumi et al. (2021) report that separating student and item networks raises simulated difficulty correlation from 0.611 to 0.886.Source: §2.3 · pp. 5–6
  • The paper extends this prior scope to 2PL discrimination, GPCM step thresholds, and NRM option slopes, whose identification differs structurally from location parameters.QIKT evaluates prediction accuracy, and Vie and Kashima (2023) compare decoder choices by test accuracy and AUC rather than parameter recovery.Source: §2.3 · pp. 5–6

3. METHODOLOGICAL FRAMEWORK

This section examines how next-response training couples item parameters, item representations, and estimated ability, including how ability changes can be absorbed without altering response probabilities. It then specifies simulated and real-data evaluations of predictive performance, parameter recovery, ranking stability, order effects, architecture comparisons, and model size.

3.1. SEQUENTIAL PREDICTION AND ITEM REPRESENTATIONS

  • SH shares one item lookup table between sequence encoding and parameter readout, whereas SK uses separate sequence and parameter tables with independently specified widths dS and dM.The response representations and encoder input operation remain unchanged; the SK parameter key is not used inside the sequence encoder.Source: §3.1 · p. 7, §3.4 · p. 12

3.2. ITEM RESPONSE MODELS AND PARAMETER READOUTS

  • The 2PL, GPCM, and NRM decode ability estimates and item parameters into response probabilities, with parameters learned as outputs of next-response training rather than separate targets.Unlike Deep-IRT’s shared item embedding for memory addressing and difficulty readout, SK supplies a separate parameter key for readout.Source: §3.2 · pp. 7, 9

3.3. PARAMETER GRADIENTS AND FISHER INFORMATION

  • Discrimination and option-slope updates depend on estimated ability and its distance from item locations, while difficulty, threshold, and intercept derivatives have no such ability term outside the response residual.This motivates evaluating slope and discrimination recovery separately from location and intercept recovery.Source: §3.3 · p. 10

3.4. GRADIENT ROUTING AND AFFINE LOGIT INVARIANCE

  • SH updates a shared item table through sequence and parameter paths, whereas SK separates those paths; item-specific ability offsets and positive gains can preserve decoder logits without proving a fixed encoder can implement them.The corresponding first-order null direction means prediction loss cannot distinguish joint ability and item-parameter changes along that direction.Source: §3.4 · pp. 11–12

3.5. EXPERIMENTAL SETUP

  • Simulations use 2,000 learners, 200 items, and 60 distinct responses per learner, generating binary, ordered, and nominal outcomes with each learner’s ability fixed across the sequence.Each learner’s generating ability follows N(0, 1).Source: §3.5 · p. 13
  • TIMSS supports item-parameter comparisons but not sequence-dynamics evaluation because item-ID ordering is not chronological; EdNet analyses retain its 250 most-answered four-option items.TIMSS includes 5,135 learners, 31 partial-credit items, and 28,324 responses; the EdNet subset includes 8,493 learners and 381,848 responses, with the same histories used for its 2PL and NRM studies.Source: §3.5 · pp. 13–14
  • Simulations compare SH and SK width choices across three encoders and response models, with 4,500 runs per item order and fixed versus independently shuffled item sequences.The fixed-order condition tiles one random item permutation; each learner receives a consecutive 60-item segment, whereas shuffled sequences independently sample 60 distinct items. Each simulated combination has 25 runs across five generated datasets and five test folds.Source: §3.5 · pp. 13–14
  • Paired architecture comparisons hold responses, train/test splits, and initialization index constant within each item-order condition.Training uses Adam for 120 epochs; simulated runs use full batches, while real-data runs use mini batches of 512 learners. Generating parameters and offline IRT estimates are reserved for evaluation.Source: §3.5 · p. 14
  • A gradient experiment contrasts unblocked and stop-gradient parameter-key readouts, which yield identical forward values but differ in gradients passed from later prediction losses.It sets dM = 64 and varies dS over {8, 16, 32, 64}; a separate DKVMN readout comparison uses width 8 and 25 paired runs per response model.Source: §3.5 · p. 14
  • Affine-channel and order-sensitivity analyses examine ability offsets and gains, item-parameter associations, and cross-order rank agreement across SH(64), SK(64, 64), and SK(64, 8).A supplementary DKVMN ablation tests whether directly passing the current item key into the ability readout induces ability–difficulty correlations under random order.Source: §3.5 · pp. 14–15
  • Evaluation covers test accuracy and log loss, simulated rank recovery of item parameters, agreement with offline IRT estimates, and cross-order ranking consistency.Offline-reference agreement uses separately fitted marginal-maximum-likelihood IRT estimates (Bock and Aitkin, 1981; Chalmers, 2012), while cross-order consistency compares learned estimates across matched administrations.Source: §3.5 · p. 15

4. RESULTS

The results examine how item order and representation width affect parameter recovery, then compare recovery with prediction accuracy and model size. Across simulations and real-data benchmarks, the effects differ for location and slope parameters and depend on how width is allocated.

4.1. ITEM ORDER AND ITEM PARAMETER ESTIMATION

  • In 2PL, fixed order associates ability offset with difficulty (ρ = −0.84) and gain with discrimination (ρ = 0.50); affine correction raises rank recovery from 0.626 to 0.934.Both offset and gain associations collapse toward zero under shuffled order.Source: Figure 2 · p. 17, §4.1 · pp. 17–18
  • Under fixed order, SK(64, 8) weakens ability–parameter associations versus SH(64), and cross-order rank agreement improves for both parameter groups in all nine pairs.SK(64, 64) does not show a consistent reduction in the associations or improvement in cross-order agreement.Source: §4.1 · p. 19, Figure 3 · p. 20

4.2. ITEM REPRESENTATION WIDTH

  • On the simulated width grid, increasing dM from 8 to 64 raises recovery at every dS, while increasing dS from 8 to 64 at dM = 64 lowers recovery in all nine pairs.The wide-key, thin-sequence corner (dM = 64, dS = 8) has the brightest tile or is near it in every Figure 5 panel.Source: §4.2 · pp. 19–20, Figure 4 · p. 21
  • SK(64, 8) raises simulated slope recovery by 0.011 to 0.186 over SH(64) across all nine pairs, with test-accuracy changes no greater than 0.008.The comparison is paired across the nine encoder–response-model combinations.Source: §4.2 · p. 20, Table 3 · p. 21
  • Blocking later-response gradients improves recovery in all nine pairs, but recovery still decreases as dS increases from 8 to 64 in every pair.None of the reductions in the width-related drop meets the predeclared rule.Source: §4.2 · pp. 22–23
  • On TIMSS and EdNet, increasing dM from 8 to 64 at dS = 8 raises offline IRT agreement in all nine pairs, while the effect of narrowing dS depends on dataset and encoder.SK(64, 8) exceeds SH(64) in seven of nine pairs, with two reversals on EdNet for the Transformer.Source: Figure 6 · p. 23, §4.2 · p. 23

4.4. TRAINABLE PARAMETERS AND MATCHED COMPARISONS

  • On EdNet 2PL and TIMSS GPCM, SK(64, 8) uses 35% to 89% as many trainable parameters as SH(64) and improves offline agreement in five of six pairs.On EdNet NRM, SK(64, 8) instead uses 107% to 109% as many parameters as SH(64).Source: Table 4 · p. 24, §4.4 · p. 25

4.3. PREDICTION ACCURACY AND ITEM PARAMETER ESTIMATION

  • In the 2PL simulation panels, test accuracy clusters around 0.71 while discrimination recovery spans roughly 0.30 at dM = 8 to above 0.90 at dM = 64.Across all nine encoder–response-model pairs, moving from SH(8) to SK(64, 8) raises recovery by 0.073 to 0.490 while test accuracy changes by less than 0.05 in every pair.Source: §4.3 · p. 24, Figure 7 · p. 25

5. DISCUSSION

Across 2PL, GPCM, and NRM decoders, accurate prediction and recovery of location parameters can coexist with poor slope recovery. The discussion traces this gap to ability-sensitive updates and item-specific ability distortions, then considers width choices, practical checks, and study limitations.

5.2. READOUT CAPACITY VERSUS SEQUENCE REPRESENTATION WIDTH

  • SK(64, 8) reduces fixed-order ability distortions and improves recovery and cross-order consistency with fewer parameters than SH(64) and no practically meaningful test-accuracy loss.On TIMSS and EdNet, widening dM from 8 to 64 at dS = 8 improves agreement with offline IRT references across all nine encoder–response-model pairs; SK(64, 8) beats SH(64) in seven of nine pairs.Source: §5 · pp. 26–27, §5.2 · p. 28

5.1. WHY PREDICTIVE ACCURACY DOES NOT VALIDATE ITEM PARAMETERS

  • Location parameters can be well recovered while discrimination and option slopes are not, because slope updates depend on ability while location updates do not.Item-specific ability offsets and gains can trade off with item parameters without changing decoder logits; in fixed-order simulations, offsets were repeatable and explained much of the uncorrected location error.Source: §5.1 · pp. 27–28

5.2. READOUT CAPACITY VERSUS SEQUENCE REPRESENTATION WIDTH

  • Separating tables at width 64 alone does not consistently improve fixed-order ability associations or cross-order rank agreement; a narrow dS limits forward history cues while dM = 64 preserves readout capacity.Blocking backward gradients also fails to eliminate the decline as dS increases, indicating the wide-sequence penalty is not solely a backward-pass artifact.Source: §5.2 · p. 28

5.3. IMPLICATIONS FOR KNOWLEDGE TRACING PRACTICE

  • For KT studies interpreting decoder parameters, evaluate parameters separately from prediction, test order sensitivity and ability shortcuts, and report parameter counts when separating dM from dS.Good recovery of difficulty, thresholds, or intercepts does not rule out errors in discrimination and option slopes.Source: §5.3 · p. 29

5.4. LIMITATIONS AND FUTURE WORK

  • Exact recovery is verified only in simulations generated from stationary unidimensional IRT models matching the decoder; real-data MML references are not independent ground truth.Fixed order tests deterministic transitions, shuffled order samples item subsets independently across learners, and TIMSS item order follows identifiers because timestamps are unavailable.Source: §5.4 · p. 29

6. CONCLUSION

The conclusion distinguishes item-parameter recovery from next-response prediction and summarizes why separating parameter-readout width from sequence width can help, while noting real-data trade-offs.

6. CONCLUSION

  • SK(64, 8) improves simulated recovery and cross-order consistency with fewer trainable parameters than SH(64) and no practically meaningful test-accuracy loss, while real-data sequence width must balance order sensitivity against predictive capacity.Across TIMSS and EdNet, widening dM consistently improves agreement with offline IRT calibrations.Source: §6 · p. 30

A.1. ADDITIONAL ITEM-ORDER COMPARISONS

The appendix presents additional item-order comparisons for GPCM and NRM and a comparison of item-estimate rankings across fixed and shuffled administrations.

A.1. ADDITIONAL ITEM-ORDER COMPARISONS

  • Figures 10 and 11 extend the item-order and affine-correction comparisons to GPCM and NRM, respectively.Figure 10 plots ϕq and sq against mean step threshold and discrimination, and varies dM in the right column with dS = 8.Source: §A.1 · p. 33, Figure 10 · p. 33
  • Figure 12 compares estimated rankings of the same 200 items across fixed and shuffled administrations for SH(64), SK(64, 64), and SK(64, 8), across all 25 matched dataset-and-fold pairs per design; larger ρordξ indicates less change.Source: §A.1 · p. 33

A.2. INDEPENDENT AFFINE CHECKS

These checks assess affine ability transformations on held-out learners and report how errors are calculated, alongside the NRM parameter panels and cross-order ranking comparison.

A.2. INDEPENDENT AFFINE CHECKS

  • The affine checks fit transformations on one half of 400 test learners and evaluate on the other, swapping halves; generating abilities are used only for comparison, with no model weights updated.The procedure covers 900 runs across SK(8,8) and SK(64,8) under both item orders.Source: §A.2 · p. 33
  • Ability error is normalized RMSE and logit error is category-centered RMSE; item-specific fits reduce ability error under fixed but not shuffled order, yet the remaining error means fits are not exact response-equivalent models.Table 5 evaluates affine-fit error on learners not used to estimate transformation coefficients.Source: §A.2 · pp. 33–34, Table 5 · p. 34
  • The NRM figure's location panels exclude 35 items with generating correct-option slope at or below 0.2, and the right column reports keyed option slope contrast.The middle column applies Eq. (25).Source: Figure 11 · p. 34, §A.2 · p. 34
  • Figure 12 reports Spearman agreement ρord ξ between item estimates from fixed and shuffled orders, with lines joining means for SH(64), SK(64, 64), and SK(64, 8).Colors identify the parameters; faint points show paired runs, while darker points average the five folds within each generated dataset.Source: Figure 12 · p. 35

A.3. DKVMN ABILITY READOUT

The supplementary DKVMN study tests whether adding the current item key to the ability readout changes ability associations and item-parameter recovery. Removing the key weakens the ability–difficulty association and changes recovery, but also slightly worsens prediction and changes parameter counts.

A.3. DKVMN ABILITY READOUT

  • The supplementary study uses five generated 2PL datasets with 800 learners, 60 items, 60 responses per learner, random balanced order, and three initialization seeds per dataset.Training uses 120 full-batch Adam epochs; the ability readout receives the memory read with or without the concatenated item key, and scores all 60 test responses.Source: §A.3 · p. 35
  • With the item key present, ability–difficulty correlations are −0.698 for SH and −0.474 for SK; removing it changes them to 0.040 and 0.110, respectively.The correlations are reported for NRM.Source: §A.3 · pp. 35–36
  • Recovery correlations exclude items with nonpositive fitted gains, although those items remain in the prediction-error check.The reported key comparison also changes readout size and does not remove item-dependent memory addressing.Source: §A.3 · p. 36
  • With the key removed, difficulty recovery rises from 0.856 to 0.977 for SH and from 0.945 to 0.983 for SK, while discrimination recovery rises from 0.669 to 0.751 for SH and falls from 0.797 to 0.779 for SK.These are NRM results; the smaller SH–SK gap accompanies weaker ability–difficulty association.Source: §A.3 · p. 36
  • Removing the key increases mean test log loss by 0.0014 in SH and 0.0010 in SK, while accuracy falls by 0.0024 and 0.0014, respectively.Parameter counts also change: SH falls from 17,651 to 15,603 and SK from 9,339 to 9,083.Source: §A.3 · p. 36

A.4. PROBABILITY QUALITY AND ITEM RANKS

The section compares prediction changes and item rankings between SH(64) and SK(64,8), using matched models and scored responses. Prediction changes are mixed across settings, and matched accuracy does not ensure equivalent probabilities or item rankings.

A.4. PROBABILITY QUALITY AND ITEM RANKS

  • The saved trajectories reproduce the original item parameters and accuracy exactly for 55 TIMSS SH(64) replays; supplementary simulated GPCM runs are not substituted for original recovery cells.The GPCM replays use five runs, one fold from each generating dataset.Source: §A.4 · p. 37
  • Mean log loss decreases in eight of nine simulated comparisons; on EdNet 2PL, SK lowers LSTM log loss by 0.076 but raises Transformer log loss by 0.068.The comparisons use matched models on the same scored responses.Source: §A.4 · p. 37, Table 7 · p. 37
  • For real-item rank comparisons, all 31 TIMSS or 250 EdNet items are included, and rank change is the mean across runs of median absolute item-rank difference.Ranks are scaled from 0 to 100 within each model and run; Figure 14 plots corresponding item ranks.Source: §A.4 · p. 37
  • The test-prediction table reports changes from SH(64) to SK(64,8), with means weighting the five generated datasets equally and dataset-level minimum and maximum log-loss changes.The GPCM comparisons use five supplementary runs per model with raw unsorted thresholds retained.Source: Table 7 · p. 37

A.5. ITEM REPRESENTATION WIDTH STUDY

The width study presents absolute simulated recovery and real-data agreement across sequence and parameter-key widths, alongside direct item-rank comparisons between SH(64) and SK(64,8). The supplied figure descriptions identify the plotted comparisons but do not state their outcomes.

A.5. ITEM REPRESENTATION WIDTH STUDY

  • Figure 14 compares item ranks assigned by SH(64) and SK(64,8), with paired runs as faint points and average item ranks as open circles; the diagonal marks equal ranks.LSTM and Transformer have 25 pairs for 2PL and GPCM and five for NRM; DKVMN has five pairs in each column.Source: Figure 14 · p. 38
  • Figure 15 presents absolute simulated recovery across dM and dS; blue lines hold dS fixed as dM varies, and the gray dashed line shows SH as w varies.The NRM row uses akey.Source: Figure 15 · p. 39
  • Figure 16 presents absolute agreement with the offline IRT reference across dM and dS on real data, using the Figure 15 encodings; its gray band shows agreement between separate response-half offline estimates.Figure 17 shows the complete SK width grid on real-data benchmarks.Source: §A.5 · p. 37, Figure 16 · p. 40, Figure 17 · p. 41

A.6. AGREEMENT ACROSS TRAINING RUNS

The supplementary material reports training-run agreement, complete matched comparisons, and trainable parameter counts. Its figure and table descriptions specify the comparisons and encodings but do not provide numerical outcomes.

A.6. AGREEMENT ACROSS TRAINING RUNS

  • Figure 18 compares item-parameter rankings across runs with different folds and initializations, averaging pairwise Spearman correlations within each SK width combination.Color limits are shared across encoders within each dataset and response model.Source: Figure 18 · p. 42, §A.6 · p. 42

A.7. COMPLETE MATCHED COMPARISONS

  • The complete matched comparisons include simulated test accuracy and recovery with 95% bootstrap intervals, plus real-data comparisons for SK(64,8).Comparisons use matching seed and fold indices; Table 10 reports cost as the parameter ratio to SH(64).Source: §A.7 · p. 42, Table 9 · p. 43, Table 10 · p. 44

A.8. TRAINABLE PARAMETER COUNTS

  • Figure 19 reports absolute recovery and offline agreement against model size, while Table 11 gives exact real-data trainable-parameter counts.In Table 11, SK holds dS = 8 while dM varies, and SH varies its shared width w.Source: Figure 19 · p. 45, §A.8 · p. 45, Table 11 · p. 46
Loading 2610.30211v1…