PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation

Hiroshi Tamano, Daichi Mochihashi

arXiv:2609.10028v1stat.MEcs.LG

TL;DR

The paper addresses accurate estimation under non-compensatory assumptions by proposing a dynamical extension of non-compensatory MIRT models. The method reproduces latent skills accurately in simulation, while the dynamical compensatory model shows significant underestimation errors.

  • Problem

    Accurate estimation requires using a correct non-compensatory model when its assumption holds, but existing dynamical extensions largely assume compensatory models.

  • Method

    The paper proposes a dynamical extension of non-compensatory MIRT models and approximates the posterior skill distribution with a Gaussian distribution by minimizing KL divergence.

  • Results

    The proposed method reproduces latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors.

  • Takeaways & Limitations

    The proposed dynamical non-compensatory MIRT model is intended to provide accurate and real-time skill tracing.

  • Takeaways & Limitations

    Preliminary experiments did not perform well, and the authors identify improving the skill transition model as a current limitation.

Abstract

from arXiv · show

Multidimensional item response theory (MIRT) is a statistical test theory that precisely estimates multiple latent skills of learners from the responses in a test. Both compensatory and non-compensatory models have been proposed for MIRT: the former assumes that each skill can complement other skills, whereas the latter assumes they cannot. This non-compensatory assumption is convincing in many tests that measure multiple skills; therefore, applying non-compensatory models to such data is crucial for achieving unbiased and accurate estimation. In contrast to tests, latent skills will change over time in daily learning. To monitor the growth of skills, dynamical extensions of MIRT models have been investigated. However, most of them assumed compensatory models, and a model that can reproduce continuous latent states of skills under the non-compensatory assumption has not been proposed thus far. To enable accurate skill tracing under the non-compensatory assumption, we propose a dynamical extension of non-compensatory MIRT models by combining a linear dynamical system and a non-compensatory model. This results in a complicated posterior of skills, which we approximate with a Gaussian distribution by minimizing the Kullback-Leibler divergence between the approximated posterior and the true posterior. The learning algorithm for the model parameters is derived through Monte Carlo expectation maximization. Simulation studies verify that the proposed method is able to reproduce latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors. Furthermore, experiments on an actual data set demonstrate that our dynamical non-compensatory model can infer practical skill tracing and clarify differences in skill tracing between non-compensatory and compensatory models.

1. Introduction

The paper motivates dynamical non-compensatory MIRT because many multi-skill tasks require all attached skills, while existing dynamical models mainly assume compensation. It proposes a model intended to trace continuous skills accurately and reports accurate recovery in simulations and practical tracing on real data.

  • Motivation: Non-compensatory models require every skill attached to a problem to exceed its threshold independently.For an equation requiring fraction and equation skills, possessing only one skill is insufficient.
  • Motivation: Compensatory models can assign high response probability when either skill is high, whereas non-compensatory models require both skills to be high.
  • Prior limitations: Non-compensatory models have been used less often because their item-response parameters are harder to estimate accurately and require many samples.Prior work reported less precise estimation than compensatory models even with many samples.
  • Prior limitations: Compensatory models significantly underestimate high skills for examinees whose actual skill levels are high in one dimension and low in another.
  • Research gap: Most dynamical models use compensatory assumptions, leaving continuous latent-skill tracing under non-compensation unresolved.The paper argues that applying compensatory dynamical models where non-compensation holds is expected to produce underestimation.
  • Contribution: The proposed dynamical non-compensatory MIRT model combines a linear dynamical system with a non-compensatory model and approximates its posterior with a Gaussian distribution.Model-parameter estimation is derived through Monte Carlo EM.
  • Results: Simulation studies show accurate latent-skill reproduction, while the dynamical compensatory model suffers from significant underestimation errors.
  • Results: Experiments on an actual dataset demonstrate practical skill tracing and clarify differences between non-compensatory and compensatory tracing.

2. Model

The model combines MIRT, which represents multiple-skill responses, with an LDS that evolves latent skills over time. Its binary observations use question-specific skill mappings and a non-compensatory emission model.

  • MIRT: MIRT estimates multiple learner proficiencies from binary responses to questions assumed to require multiple skills.
  • MIRT: Compensatory MIRT sums skill contributions, whereas non-compensatory MIRT multiplies them so individual skills cannot complement one another.
  • Linear Dynamical System: An LDS models latent states that transition from the previous state and emit observable data, with inference using Kalman smoothing and EM.
  • Dynamical model: The proposed dynamical non-compensatory MIRT uses binary, question-indexed observations and a non-compensatory item-response emission within an LDS-like generative model.
  • Dynamical model: Each learner’s latent state is a K-dimensional skill vector that begins Gaussian, undergoes linear transitions, and generates Bernoulli responses.
  • Dynamical model: The Q-matrix indicates which skills each question requires, and the model assumes this mapping is known in advance.
  • Dynamical model: Covariates can adapt transitions to factors such as elapsed time since practice, including a forgetting factor based on elapsed time and memory half-life.
  • Model variants: Fixing discrimination parameters to one yields an LDS–multicomponent latent-trait combination that is sufficient to infer skills in most cases.

3. Inference

Inference follows the LDS forward-backward and EM framework, but the non-compensatory likelihood is non-conjugate with the Gaussian prior. The paper therefore approximates the forward posterior message as Gaussian.

  • Inference framework: The method estimates skill-state posteriors and model parameters using LDS-based forward-backward and EM algorithms.
  • Message passing: Forward message passing and backward message passing are used to calculate the posterior of each latent state.
  • Inference challenge: Unlike ordinary LDS inference, the forward message cannot be calculated exactly because the non-compensatory likelihood is non-conjugate with the Gaussian prior.
  • Inference solution: The proposed solution approximates the forward message with a Gaussian distribution before applying the remaining inference and parameter-estimation steps.

3.1. Gaussian Approximation of ˆα Message

The Gaussian approximation chooses a posterior distribution by minimizing KL divergence to the true posterior. The resulting objective is optimized using reparameterized sampling and gradients of the approximation parameters.

  • Approximation objective: The approximation seeks q(z)=N(z|µ,V) that minimizes KL divergence between the approximated posterior and p(z|y).
  • Approximation objective: The KL objective combines divergence from the prior with the expected log likelihood under the approximated posterior.
  • Optimization: A reparameterization trick approximates the intractable expected log likelihood by sampling from the approximated posterior.
  • Optimization: The covariance is represented through its Cholesky factor L, satisfying V=LL^⊤, and gradients are derived with respect to µ and L.
  • Optimization: Gradient-based optimization finds the Gaussian parameters µ and L, using gradient descent with Armijo line search in the implementation.

3.2. E-step

The E-step infers skill-state posteriors from response sequences using forward-backward message passing, with Gaussian approximations for otherwise complicated distributions.

  • The E-step infers each skill-state posterior given model parameters and the learner’s observed response sequence.
  • Forward and backward messages are calculated across the time series to obtain posteriors for all skill states.
  • The forward message separates skills required by the current question from the remaining skills before applying Gaussian approximation.
  • Forward messages are computed recursively from time step 1 through T_j, with each message approximated as a Gaussian.
  • The backward γ message is passed backward through the sequence to obtain posteriors for all learner skill states.

3.3. M-step

The M-step maximizes the expected complete-data log-likelihood, separately updating initial-state, transition, and emission parameters while using priors to limit overfitting.

  • The M-step maximizes the expected complete-data log-likelihood with respect to the model parameters.
  • Priors are introduced for model parameters to avoid overfitting, including Gaussian-Wishart priors for initial-state parameters.
  • Optimization is divided into initial-state, state-transition, and emission parameter updates.
  • The state transition uses a question-specific linear transformation whose repeated application approaches β_k, i_k/(1 − d_i(j,t),k) when 0 < d_i(j,t),k < 1.
  • A joint prior on transition slope and bias parameters constrains inferred skills within a reasonable range by incorporating prior knowledge of asymptotic skill values.
  • Emission parameters are optimized using sampled expectations and gradient-based optimization under log-normal and Gaussian priors.

3.4. Smoothing Extension in E-step

The smoothing extension broadens each time step’s likelihood with nearby observations, allowing local temporal information to contribute to Gaussian skill-state inference.

  • Smoothing extends the E-step by incorporating observations near the time step being inferred.
  • The local likelihood is replaced by a weighted product of likelihoods from adjacent time steps.
  • The Gaussian approximation is derived as before after replacing the original likelihood with the smoothed likelihood.
  • The evaluation experiments smoothing with a ±1 time-step width.

4. Evaluation of Skill Inference

Simulation experiments compare the proposed dynamical non-compensatory model with dynamical compensatory and static non-compensatory alternatives across six data settings. The proposed model generally achieves lower skill-estimation error, while the compensatory model systematically underestimates skills.

  • Experimental design: The evaluation compares dnMIRT, dcMIRT, and nMIRT across six simulated data types varying skill dimensionality, discrimination parameters, and initial-state correlation.
  • Skill inference results: dnMIRT with smoothing generally produces the best MAE, while dcMIRT has large MAEs despite high correlations.
  • Skill inference results: dcMIRT underestimates skills, whereas nMIRT incurs large errors because it does not model latent-skill changes over time.
  • Parameter settings: Estimating discrimination parameters increases skill and parameter errors relative to settings where discrimination is fixed.
  • Temporal behavior: Smoothing lowers error near the end of sequences, although unsmoothed dnMIRT can perform better at the beginning.
  • Temporal behavior: Skill-estimation MAEs decrease over time and saturate after approximately ten time steps for two-skill data and fifty time steps for 100-skill data.
  • Temporal behavior: Approximately five logs of answered questions per skill are required to obtain stable skill estimates.

5. Evaluation of Prediction

The evaluation compares dynamical non-compensatory MIRT (dnMIRT) with dynamical compensatory MIRT (dcMIRT) for skill inference and next-answer prediction. Across simulated settings, dnMIRT generally better preserves skill estimates and avoids the underestimation associated with dcMIRT.

  • Prediction: dnMIRT outperformed dcMIRT on three prediction metrics for both data sets D and F.The advantage was larger for predicting incorrect answers than correct answers.
  • Prediction: Prediction models with higher average precision for incorrect answers are useful because incorrect-answer alerts are more common in the intended application.The evaluation treated learner response sequences as sequential binary classification and used forward messages to predict each next answer.
  • Skill inference: dnMIRT reproduced the true skills adequately, whereas dcMIRT tended to underestimate skills despite high correlation.The dcMIRT response-surface contour was below the true model, producing significant skill underestimation.
  • Prediction: The prediction performance was not sensitive to the tested discrimination-parameter settings, while the dcMIRT gap was somewhat smaller when initial skills were correlated.Data sets D and F differed in whether initial skills correlated.

6. Evaluation of ˆα Message Approximation

The study evaluates a Gaussian approximation to the posterior message used by the proposed method against the Laplace approximation. The proposed approximation better follows complex posteriors and has lower KL-divergence in the tested cases, while its error decreases with more samples before saturation.

  • Skill estimation: Skill-estimation errors saturated at approximately ten time steps for two-skill data sets and fifty time steps for 100-skill data sets.This result was reported for data sets A and B versus data sets C and D.
  • Posterior shape: The proposed posterior adequately approximated the true posterior for both correct and incorrect answers.For incorrect answers, it covered two posterior peaks that the Laplace approximation missed and correctly represented other complex shapes.
  • Approximation error: The Laplace approximation misrepresented posterior variance and became less accurate away from the posterior maximum.It underestimated one variance direction, overestimated another, and extended into an incorrect region in the examined cases.
  • Approximation error: The proposed method had smaller KL-divergence than the Laplace approximation for both correct and incorrect answers.The KL-divergence increased with dimension for correct answers but decreased with dimension for incorrect answers.
  • Sample size: KL-divergence decreased as the number of samples increased and saturated at approximately 100 samples.The simulations used 50 samples for skill inference as a trade-off between approximation error and computation cost.

7. Application to Real Data

The real-data study applies dnMIRT to ASSISTments learning records for next-answer prediction and longitudinal skill tracing. dnMIRT slightly but significantly improves prediction over dcMIRT and yields skill changes that distinguish which skills explain individual responses.

  • Joint prior: The joint Gaussian prior constrains slope and bias parameters so inferred asymptotic skills remain approximately between one and two.The prior was introduced because unconstrained models could overfit and infer implausibly large or small skills.
  • Prediction: dnMIRT was significantly better than dcMIRT on three prediction metrics, although the gap was small.Five-fold cross-validation was used, and Wilcoxon signed-rank tests verified differences at a 5% confidence level.
  • Skill tracing: dnMIRT attributed individual responses to specific skill changes, whereas dcMIRT often changed multiple skills together.For one learner’s incorrect answer, dnMIRT decreased “Box and Whisker” more than “Range,” while dcMIRT decreased both skills.
  • Skill tracing: In dnMIRT, the direction of a skill change depended on the current skill state because its response contour was nonlinear.The model inferred different changes for related skills across learners and response events.
  • Skill tracing: For learner D, “Multiplication and Division Integers” grew before the other three tracked skills, which grew later.The longer trace illustrates differentiated temporal growth across skills.

8. Discussion

The proposed dynamical non-compensatory MIRT model combines a linear dynamical system with non-compensatory MIRT and uses Gaussian posterior approximation with Monte Carlo EM. Simulations and actual-data experiments support accurate skill tracing, improved prediction over the compensatory counterpart, and practical differences in inferred skill trajectories, while highlighting several scope and modeling limitations.

  • Model and inference: The model combines a linear dynamical system with non-compensatory MIRT and estimates parameters through Monte Carlo expectation maximization.Its posterior is approximated as a Gaussian by minimizing KL divergence from the true posterior.
  • Simulation studies: Simulation studies reproduced true latent skills across 2-skill and 100-skill settings, whereas the dynamical compensatory model suffered significant underestimation errors.The proposed approach remained effective as latent-skill dimensionality increased to 100.
  • Approximation accuracy: The Gaussian approximation was better than the Laplace approximation at approximating the true posterior in the reported quantitative analyses.The comparison evaluated approximation error using visualizations and quantitative analyses.
  • Actual-data experiments: On actual data, the model inferred practical skill tracing and clarified how non-compensatory and compensatory models differed in attributing skill changes.The non-compensatory model produced skill trajectories that differed from those of the compensatory counterpart across learners and time steps.
  • Prediction and deployment: Prediction accuracy was slightly better than the compensatory version, while inference after training took 0.0052 sec for 100-skill data.Training 100-skill data took about one to two days on a 36-core machine under one hyperparameter setting.
  • Limitations and future work: The study is limited by linear skill transitions, incomplete exploration of smoothing and weight-parameter combinations, and evaluation on only one actual data set.The authors also identify exploratory-analysis extension and broader actual-data comparisons as future work.

Appendix

The appendix lists the first 30 skill tags in ASSISTments2009-2010 and reports standard deviations for skill-inference accuracy across six datasets. It also contains entries covering geometry, probability, statistics, and measurement topics.

  • Table 10 lists the first 30 skill tags in the ASSISTments2009-2010 skill builder dataset.
  • Additional entries cover polygon geometry, probability, statistics, graphs, number lines, conversions, and related mathematics topics.
  • Tables 11–16 report standard deviations in skill-inference accuracy for datasets A–F under varying skill counts, discrimination distributions, and skill-correlation settings.
Loading 2609.10028v1…