PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation

Hiroshi Tamano, Daichi Mochihashi

arXiv:2609.10028v1stat.MEcs.LG

TL;DR

Most dynamical MIRT models assume compensatory skills, leaving continuous skill states under a non-compensatory assumption unaddressed. This paper combines a linear dynamical system with a non-compensatory MIRT model and reports accurate skill reproduction in simulations, while the dynamical compensatory model significantly underestimates skills.

  • Problem

    Most dynamical MIRT models assume compensatory skills, and a model reproducing continuous skill states under a non-compensatory assumption had not been proposed.

    Source: §1 · p. 5

  • Method

    The paper combines a linear dynamical system with a non-compensatory MIRT model and approximates the skills posterior with a Gaussian distribution using KL divergence minimization.

    Source: §1 · p. 8

  • Results

    Simulations show that the proposed method reproduces latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors.

    Source: §1 · p. 8, §8 · p. 65

  • Takeaways & Limitations

    Experiments on actual data show that the model can infer practical skill tracing and clarify differences between non-compensatory and compensatory skill traces.

    Source: §8 · p. 61

    The authors identify preliminary experiments that did not perform well as a current limitation.

    Source: §8 · p. 69

Abstract

from arXiv · show

Multidimensional item response theory (MIRT) is a statistical test theory that precisely estimates multiple latent skills of learners from the responses in a test. Both compensatory and non-compensatory models have been proposed for MIRT: the former assumes that each skill can complement other skills, whereas the latter assumes they cannot. This non-compensatory assumption is convincing in many tests that measure multiple skills; therefore, applying non-compensatory models to such data is crucial for achieving unbiased and accurate estimation. In contrast to tests, latent skills will change over time in daily learning. To monitor the growth of skills, dynamical extensions of MIRT models have been investigated. However, most of them assumed compensatory models, and a model that can reproduce continuous latent states of skills under the non-compensatory assumption has not been proposed thus far. To enable accurate skill tracing under the non-compensatory assumption, we propose a dynamical extension of non-compensatory MIRT models by combining a linear dynamical system and a non-compensatory model. This results in a complicated posterior of skills, which we approximate with a Gaussian distribution by minimizing the Kullback-Leibler divergence between the approximated posterior and the true posterior. The learning algorithm for the model parameters is derived through Monte Carlo expectation maximization. Simulation studies verify that the proposed method is able to reproduce latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors. Furthermore, experiments on an actual data set demonstrate that our dynamical non-compensatory model can infer practical skill tracing and clarify differences in skill tracing between non-compensatory and compensatory models.

1. Introduction

The paper introduces a dynamical non-compensatory MIRT model to trace continuous skills, addressing a gap in existing dynamical models. Simulations and real-data experiments support its skill-tracing utility and show differences from compensatory tracing.

Introduction

  • Non-compensatory models require each skill to exceed its threshold independently, fitting multi-skill tasks tagged with “and” conditions where all attached skills must be mastered.Source: §1 · p. 4
  • Accurate non-compensatory item-parameter estimation has been difficult: Bolt (2013) reported that MCMC required many samples yet remained less precise than compensatory estimation.Source: §1 · p. 5
  • Buchholz and Hartig (2018) found compensatory models significantly underestimated high skills for examinees high in one skill but low in another, potentially disadvantaging this subgroup.Source: §1 · pp. 5–6
  • Most dynamical models are compensatory, while a model reproducing continuous latent skills under the non-compensatory assumption had not been proposed; prior cognitive-diagnostic dynamics used binary mastery states.Source: §1 · p. 7
  • The proposed model combines a linear dynamical system with non-compensatory MIRT, approximates the complex skill posterior with a Gaussian via KL minimization, and estimates parameters using Monte Carlo EM.Source: §1 · p. 8
  • In simulations, the proposed model reproduced latent skills accurately while the dynamical compensatory model significantly underestimated them; real-data experiments yielded practical tracing and model-specific differences.Figure 2 visualizes skill tracing for a learner on the ASSISTments 2009–2010 dataset.Source: §1 · p. 8, Figure 2 · p. 9

2. Model

The model combines non-compensatory MIRT with a linear dynamical system to trace learners’ multiple skills over time. Its formulation uses binary response data and a question–skill mapping assumed known in advance.

2.1. Multidimensional Item Response Theory

  • MIRT estimates examinees’ proficiency and question difficulty across multiple skills from binary correct-or-incorrect responses, with questions assumed to require multiple skills.Source: §2.1 · p. 10

2.2. Linear Dynamical System

  • An LDS infers changing latent states from noisy observations, with Gaussian initial states, linear transitions, and emissions; a Kalman smoother estimates states and EM estimates parameters.Source: §2.2 · pp. 11–12

2.3. Dynamical Non-compensatory MIRT

  • The proposed model extends non-compensatory MIRT with linear Gaussian state transitions and a non-compensatory item-response emission, using a Q-matrix assumed known in advance.Learner skill states begin with a Gaussian distribution and evolve through linear transitions; each question’s response is modeled with a non-compensatory item-response model.Source: §2.3 · pp. 12–14
  • Fixing discrimination parameters ai,k to one yields an LDS–multicomponent latent trait model combination that is sufficient to infer skills in most cases.Source: §2.3 · p. 15

3. Inference

Inference combines an LDS-style forward-backward procedure for skill-state posteriors with EM estimation of model parameters. Because the non-compensatory likelihood is not conjugate to the Gaussian prior, the forward message is approximated as Gaussian.

3. Inference

  • The inference methods use forward-backward estimation for skill-state posteriors and EM for model parameters.These methods are based on the corresponding LDS algorithms.Source: §3 · p. 15
  • Non-conjugacy between the non-compensatory likelihood and Gaussian prior prevents exact calculation of the forward message, so it is approximated as Gaussian.After this approximation, the backward message is calculated using the LDS method.Source: §3 · p. 15

3.1. Gaussian Approximation of ˆα Message

The section approximates the skill-state posterior with a Gaussian distribution by minimizing its KL divergence from the posterior formed by a non-compensatory item-response likelihood and a Gaussian prior.

3.1. Gaussian Approximation of ˆα Message

  • The α̂ message combines a non-compensatory item-response likelihood and Gaussian prior; its posterior is approximated by a Gaussian q(z) minimizing KL divergence to the true posterior.Source: §3.1 · p. 16
  • The KL objective comprises divergence between the approximating posterior and prior, minus the approximating posterior’s expected log likelihood.Source: §3.1 · p. 16
  • Because the expected log likelihood is intractable, the method uses the reparameterization trick (Kingma and Welling 2013), sampling through a Cholesky factor of the covariance.Source: §3.1 · pp. 16–17
  • Gradients with respect to the Gaussian parameters enable gradient-based optimization of its mean and Cholesky factor; the implementation uses gradient descent with Armijo line search.Source: §3.1 · p. 17

3.2. E-step

The E-step estimates latent-skill posteriors from model parameters and observed responses using forward and backward messages. Its forward pass uses a Gaussian approximation, while the backward message follows the LDS result.

3.2. E-step

  • Forward and backward message passing combines information across the time series to obtain posterior distributions for all skill states.The E-step is conditioned on the model parameters and learner response sequence.Source: §3.2 · p. 17

3.2.1. Forward Message Passing

  • The forward pass recursively computes Gaussian-approximated α messages from time 1 through T_j, separating skills required by the current item from other skills.The Gaussian approximation is applied to p(y^(t)_j|z_a)p(z_a).Source: §3.2.1 · p. 18

3.2.2. Backward Message Passing

  • Because the forward α message is Gaussian, the backward γ message matches the LDS result and is passed backward to obtain posteriors for all skill states.The derivation is omitted in the passage, which refers to Bishop (2006).Source: §3.2.2 · pp. 19–20

3.3. M-step

The M-step maximizes the expected complete-data log-likelihood over model parameters, using priors and separating the optimization into initial-state, state-transition, and emission parameters.

3.3. M-step

  • The expected complete-data log-likelihood is maximized over model parameters, with priors introduced to avoid overfitting and optimization divided among initial-state, transition, and emission parameters.The section denotes expectations with respect to the latent skill trajectories.Source: §3.3 · p. 20

3.3.1. Initial State

  • The initial-state parameters µ0 and P0 receive a Gaussian-Wishart prior.The section formulates an objective for these parameters and describes setting its derivative to zero for optimization.Source: §3.3.1 · p. 21

3.3.2. State Transition

  • State-transition parameters use Gaussian-gamma priors, and the transition probability is Gaussian with diagonal covariance.The optimization is independent across parameters except for γk.Source: §3.3.2 · pp. 21–22

3.3.3. State Transition with a Joint Prior on Slope and Bias

  • A joint Gaussian prior on slope and question-specific bias encodes prior knowledge of asymptotic skill levels to help keep inferred skills within a reasonable range.With slope between 0 and 1, the skill asymptotically approaches the bias divided by one minus the slope when the question is repeatedly answered.Source: §3.3.3 · p. 25

3.3.4. Emission

  • Emission parameters ai,k and bi,k receive log-normal and Gaussian priors, respectively, and gradient-based optimization obtains their optimal values.The expectation in the objective is approximated by sampling.Source: §3.3.4 · p. 27

3.4. Smoothing Extension in E-step

The E-step is extended with smoothing so observations near time t can help infer the skill state at t. The method combines likelihoods from neighboring time steps and evaluates a window of width ±1.

3.4. Smoothing Extension in E-step

  • Observations near time t, including those at t−1 and t+1, can provide information for inferring the skill state at t.Source: §3.4 · p. 28
  • The E-step is extended with a smoothing technique using weighted likelihood products for skills required by questions at t−1, t, and t+1; the α̂ message’s Gaussian approximation is derived as in Section 3.1.Source: §3.4 · p. 28
  • The evaluations examine the effect of smoothing with a window of width ±1.Source: §3.4 · p. 28

4. Evaluation of Skill Inference

The evaluation examines skill-inference accuracy across six simulated data types, comparing the proposed dynamical non-compensatory model with dynamical compensatory and non-dynamical non-compensatory MIRT. It reports that dnMIRT generally achieved lower skill-estimation MAEs than dcMIRT, with MAEs decreasing over time before saturating.

4.1. Simulation Design

  • Six simulation-data types varied three aspects: latent-skill count (2 or 100), whether generative-model discrimination parameters were fixed to one or randomly generated, and whether initial-state skills were correlated; five datasets were used per type.Source: §4.1 · p. 29, §4.2 · p. 29

4.3. Inference Settings

  • When estimating γk, the authors used validation data to select the EM stopping point because γk estimation tended to overfit; smoothing was not combined with γk estimation after poor preliminary results.Validation users numbered 20% of the training users, and stopping was based on next-response prediction accuracy.Source: §4.3 · p. 32

4.4. Results

  • dnMIRT had much smaller skill MAEs than the other models in data set A; dcMIRT had the worst MAE, while nMIRT was significantly inferior to dnMIRT.The reported nMIRT disadvantage was attributed to difficulty-parameter estimation error and its failure to model changing skills.Source: §4.4 · p. 33
  • Smoothing reduced dnMIRT’s MAE in all data types except E, and fixing γk yielded slightly better MAE than estimating it in data set A.The authors attributed the fixed-γk advantage to using a near-optimal value.Source: §4.4 · p. 33
  • On data set A, dnMIRT had small MAEs and high correlations, whereas dcMIRT had large MAEs and high correlations and nMIRT had large MAEs and low correlations.dcMIRT’s response-surface contours were below the true-model contours, a fitting trend the authors associate with its skill underestimation.Source: §4.4 · p. 34
  • Across dnMIRT settings, fixing discrimination at a_i,k = 1 with smoothing yielded the best MAE; estimating γ_k gave better MAE than fixing it, while σ_a = 0.25 without smoothing was worst.Adding discrimination parameters to be estimated resulted in larger skill and parameter errors.Source: §4.4 · pp. 34–35
  • On data set C, data set A’s result trends persisted, and the skill-inference correlation gap between dnMIRT and dcMIRT became large; nMIRT results were unavailable because its 100-skill estimation required significant memory.Source: §4.4 · p. 35
  • Smoothing did not reduce skill-inference MAE, and dnMIRT with estimated γ_k had worse MAE despite high correlation; relative to D, data set E had better skill-inference MAE but worse correlation.Source: §4.4 · pp. 35–36
  • Standard deviations were small for almost all results; data set B’s skill-inference MAEs had relatively large standard deviations, although its correlations had small standard deviations.Source: §4.4 · p. 36

5. Evaluation of Prediction

On simulation datasets D and F, dnMIRT outperformed dcMIRT on all three prediction metrics, with a larger advantage for predicting incorrect answers. Prediction performance was not sensitive to discrimination-parameter settings, and the advantage was slightly smaller on F than D.

5. Evaluation of Prediction

  • Evaluation used 5-fold cross-validation on datasets D and F, with 80% of learner logs for training and 20% for testing.Source: §5 · p. 37
  • Test responses were predicted sequentially with forward messages, excluding each learner’s first log as in Piech et al. (2015); metrics were AUC and average precision for correct and incorrect answers.Source: §5 · p. 37
  • On datasets D and F, dnMIRT outperformed dcMIRT on AUC and both average-precision metrics, with a larger gap for incorrect-answer prediction.The larger gap indicates dnMIRT was particularly better at predicting incorrect answers.Source: §5 · p. 37
  • High average precision for incorrect answers is useful because informing users which questions they may answer incorrectly is more common.Source: §5 · p. 37
  • Prediction performance was not sensitive to discrimination-parameter settings, and the dnMIRT–dcMIRT gap was slightly smaller for correlated-initial-skill dataset F than for D.Datasets D and F differ in whether initial-state skills correlate.Source: §5 · p. 45

6. Evaluation of ˆα Message Approximation

The evaluation examines whether the Gaussian approximation captures the true posterior, how its KL-divergence compares with the Laplace approximation across latent dimensions, and how sample count affects approximation error.

6.1. Visual Evaluation of Approximated Posterior

  • For correct answers, the proposed approximation matches the true posterior in the illustrated cases, while Laplace underestimates x-axis variance and overestimates y-axis variance in case (a).Laplace also extends to the lower left in case (b), with errors tending to grow farther from the posterior maximum.Source: §6.1 · p. 48
  • For incorrect answers, whose true posteriors are more complex, the proposed method covers both peaks in case (d) and correctly covers the posterior in cases (e) and (f), unlike Laplace.Laplace covers only one peak in case (d) and overestimates variance in cases (e) and (f).Source: §6.1 · pp. 48–49

6.2. Quantitative Evaluation of Approximated Posterior

  • Across correct and incorrect answers, the proposed method has lower KL-divergence than the Laplace approximation; error rises with dimension for correct answers but falls for incorrect answers.The incorrect-answer trend is attributed to a decreasing intersection between the likelihood hyper-cube and prior hyper-sphere as dimensions increase.Source: §6.2 · pp. 49, 52, Figure 9 · p. 52

6.3. Relationship between Approximation Error and Sample Size

  • KL-divergence decreases as the number of approximation samples increases and saturates at about 100 samples; skill-inference simulations used 50 to balance error and computation cost.The pattern holds for both correct and incorrect answers.Source: Figure 10 · p. 53, §6.3 · p. 53

7. Application to Real Data

The real-data application uses ASSISTments 2009–2010 data to evaluate dynamical models for next-question prediction and visualize learner skill tracing. It examines differences in prediction and in how dnMIRT and dcMIRT represent skill changes over time.

7.1. Data

  • After filtering scaffolding questions and questions answered by fewer than ten learners, the data comprised 224,905 records, 4,106 learners, 8,112 questions, and 106 skills.Source: §7.1 · p. 54

7.2. Joint Prior on Slope and Bias

  • A joint prior on slope and bias constrained asymptotic skill to approximately 1–2; without it, both models overfit and inferred implausibly extreme skill values.The authors constructed a Gaussian prior that approximately covers the slope-and-bias region associated with asymptotic skills from one to two.Source: §7.2 · p. 55

7.3. Performance of Prediction

  • dnMIRT slightly outperformed dcMIRT on all three metrics, with statistically significant but small differences; smoothing differed on AUC and incorrect-answer precision (Table 9).The Wilcoxon signed-rank test found differences between dnMIRT and dcMIRT at the 5% confidence level; dnMIRT was significantly better. The best result for each model used the same hyperparameter settings.Source: §7.3 · p. 57, Table 9 · p. 58

7.4. Skill Tracing

  • For learner A, dnMIRT reduced “Box and Whisker” more than “Range” after an incorrect answer, while dcMIRT reduced both; dnMIRT’s skill-change direction depends on current state.dnMIRT inferred “Box and Whisker” as the cause of the incorrect response, whereas dcMIRT decreased both skills.Source: §7.4 · pp. 58–59
  • For learner B, dnMIRT inferred weak Rotations and Translations skills and decreased both to explain the incorrect answer at step 37; dcMIRT also explained the error.Both skills decreased from step 36 to step 37.Source: §7.4 · p. 60
  • For learner C, dnMIRT increased Unit Conversion while Addition and Subtraction changed little, whereas dcMIRT increased both skills.dnMIRT estimated Unit Conversion as weaker and attributed the correct answer to its larger increase.Source: §7.4 · pp. 60–61
  • In learner D’s longer trace, Multiplication and Division Integers grew first, while Absolute Value, Subtraction Whole Numbers, and Addition Whole Numbers grew later.The latter three skills were more difficult for learner D to acquire.Source: §7.4 · p. 61

8. Discussion

The paper proposes a dynamical non-compensatory MIRT model for skill tracing and evaluates it through simulations and real-data experiments. The discussion considers its prediction accuracy on an actual dataset as well as limited empirical evidence and unresolved modeling limitations.

8. Discussion

  • The model combines an LDS with non-compensatory MIRT, approximates the complicated posterior by a Gaussian through KL-divergence minimization, and estimates parameters using Monte Carlo EM.Source: §8 · p. 61
  • On actual data, the model produced practical skill traces, attributed success or failure to particular skills, and was slightly better than its compensatory version at predicting responses.The discussion reports that Figure 2 shows practical tracing.Source: §8 · p. 65
  • Unlike SPARFA-Trace (Lan et al. 2014), which supports exploratory Q-matrix inference, this confirmatory method assumes a given Q-matrix; its variational approximation applies to both compensatory and non-compensatory models.Extending the proposed model to exploratory analysis remains future work.Source: §8 · pp. 65–66
  • After training, forward message passing inferred current skills from each response in 0.0052 seconds for 100-skill data, and the method also worked at 100 dimensions.Source: §8 · pp. 66–67
  • The Gaussian approach scales to high-dimensional skills, whereas sampling can express arbitrary posterior shapes but functions only in low dimensions, motivating comparisons of accuracy, speed, and cost.Source: §8 · p. 67
  • The model can be reformulated for longitudinal IRT by accepting multiple responses per time step, and the authors recommend fixing discrimination parameters to one, using a joint slope-bias prior, and avoiding smoothing initially.Source: §8 · pp. 67–68
  • The linear transition can decrease a high-skill learner’s skill after correctly answering an easy problem; suggested alternatives include a piecewise-linear rule or Gaussian-process transitions (Wang et al. 2005).Source: §8 · p. 68
  • The dynamical non-compensatory MIRT model was slightly more accurate than its compensatory version on one actual dataset.The authors call for comparisons across various actual datasets to determine whether this advantage generally holds.Source: §8 · pp. 68–69
  • The models’ comparative performance remains uncertain because the comparison used only one actual dataset.The authors say comparisons on various actual datasets are required to assess whether the non-compensatory model generally performs better.Source: §8 · p. 69
  • Heuristics set a joint prior on skill-transition slopes and biases; without it, both models overfit and test accuracy declined after iterations.The authors manually tried heuristics and chose the best-performing one.Source: §8 · p. 69
  • An unexpectedly large or small skill value could not be measured by item difficulty parameters distributed as N(0, 1).The authors expect further studies to improve the skill-transition model so it can be used without setting the joint prior.Source: §8 · p. 69
  • The authors did not combine the smoothing extension with estimation of γk because preliminary experiments performed poorly.They identify this as a current limitation and say further exploration of weight parameters is needed to confirm whether the combination truly fails.Source: §8 · p. 69

Appendix

The appendix identifies the first 30 skill tags in ASSISTments2009-2010 skill builder and provides tables of standard deviations in skill-inference accuracy under several simulated data conditions.

Appendix

  • Table 10 identifies the first 30 skill tags in ASSISTments2009-2010 skill builder.Source: Table 10 · p. 70
  • Tables 11–14 report standard deviations in skill-inference accuracy for data sets A–D, spanning 2-skill and 100-skill settings with fixed or log-normal item parameters.The captions specify no skill correlation for A and C; they describe B and D as using logN(0, 0.25^2) item parameters.Source: Table 11 · p. 71, Table 12 · p. 72, Table 13 · p. 73
  • Tables 15–16 report standard deviations in skill-inference accuracy for 100-skill data sets E and F with different log-normal item-parameter settings.The captions give logN(0.4, 0.2) for E and logN(0, 0.25) for F; the latter caption also mentions skill correlation.Source: Table 15 · p. 75, Table 16 · p. 76

Related papers

Loading 2609.10028v1…