PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep-IRT: Make Deep Learning Based Knowledge Tracing Explainable Using Item Response Theory

Chun-Kit Yeung

arXiv:1904.11738v1cs.LGcs.AIstat.ML

TL;DR

Deep learning based knowledge tracing can predict effectively but remains difficult to interpret. Deep-IRT combines DKVMN with IRT to estimate student ability and item difficulty over time, and experiments show it retains DKVMN’s performance while aligning difficulty estimates with traditional methods. The model still exhibits reconstruction-related and learning-behavior limitations.

  • Problem

    Deep learning based knowledge tracing models can be difficult to interpret, unlike structured models whose parameters have direct psychological meaning.

  • Method

    Deep-IRT uses DKVMN to process learning trajectories and estimate student ability and KC difficulty, then uses IRT to predict correct-response probability.

  • Results

    Deep-IRT retains DKVMN’s performance, while its estimated KC difficulty aligns with traditional IRT and item-analysis methods.

  • Takeaways & Limitations

    Deep-IRT provides psychologically meaningful estimates of student ability and KC difficulty while preserving the predictive power of deep learning based knowledge tracing.

  • Takeaways & Limitations

    Deep-IRT still exhibits reconstruction-related issues and counterintuitive ability and prediction changes, including difficulty modeling prerequisite relationships among skills.

Abstract

from arXiv · show

Deep learning based knowledge tracing model has been shown to outperform traditional knowledge tracing model without the need for human-engineered features, yet its parameters and representations have long been criticized for not being explainable. In this paper, we propose Deep-IRT which is a synthesis of the item response theory (IRT) model and a knowledge tracing model that is based on the deep neural network architecture called dynamic key-value memory network (DKVMN) to make deep learning based knowledge tracing explainable. Specifically, we use the DKVMN model to process the student's learning trajectory and estimate the student ability level and the item difficulty level over time. Then, we use the IRT model to estimate the probability that a student will answer an item correctly using the estimated student ability and the item difficulty. Experiments show that the Deep-IRT model retains the performance of the DKVMN model, while it provides a direct psychological interpretation of both students and items.

1. INTRODUCTION

Knowledge tracing models student knowledge from learning interactions, but the field faces a trade-off between interpretable structured models and powerful deep models with difficult-to-interpret representations. Deep-IRT addresses this trade-off by combining DKVMN with IRT to estimate psychologically meaningful student and item parameters while retaining predictive performance.

  • Knowledge tracing models student knowledge state from historical interactions to predict aspects of a student’s next interaction.Question-and-answer interactions are commonly represented by a question tag and an answer-correctness label.
  • Structured models such as BKT and PFA offer directly meaningful parameters, whereas DKT and DKVMN are complex, general-purpose models that typically perform better without substantial human feature engineering but are difficult to interpret.
  • Deep-IRT combines a deep learning model with a psychometric model to make deep learning based knowledge tracing explainable.
  • DKVMN processes historical question-and-answer interactions to infer student ability and KC difficulty over time, after which IRT predicts the probability of a correct response.
  • Deep-IRT retains DKVMN’s performance while providing meaningful estimates of student ability and KC difficulty.
  • The model potentially estimates KC difficulty from entire learning trajectories rather than a traditional educational testing environment, and its deep-to-psychometric design may apply beyond knowledge tracing.

2. LITERATURE REVIEW

Knowledge tracing spans psychologically interpretable structured models and more complex deep models, with IRT providing ability–difficulty interpretations but assuming static ability. Deep models avoid extensive feature engineering and can model richer trajectories, yet their representations remain difficult to interpret.

  • 2.1 Item Response Theory: IRT estimates the probability of a correct response from student ability θ and item difficulty βj using a logistic response function.The one-parameter formulation is P(a) = σ(θ − βj).
  • 2.1 Item Response Theory: IRT cannot directly model knowledge tracing because it assumes that student ability remains constant during a test.Knowledge tracing instead models changing student knowledge states over time.
  • 2.2.2 Factors Analysis Based Knowledge Tracing: LFA and PFA use logistic regression with skill-level parameters, while PFA separates successful and failed attempts to increase sensitivity to performance.LFA uses attempt counts Nj; PFA uses Sj and Fj and performs better than LFA.
  • 2.2.2 Factors Analysis Based Knowledge Tracing: LFA and PFA require manual skill labels and use relatively simple human-engineered features that may inadequately represent student knowledge states.Their formulations also impose constraints on how response probability changes after repeated attempts.
  • 2.2.3 Deep Learning Based Knowledge Tracing: DKT uses an RNN to summarize a student’s past interactions in a latent knowledge state and has outperformed traditional models without human-engineered features.Its hidden state is computed from sequential interaction vectors, including with LSTM cells.
  • 2.2.3 Deep Learning Based Knowledge Tracing: DKVMN distributes latent concepts and knowledge states across static key and dynamic value memories, but its vector representations remain difficult to interpret.The model uses attention to combine latent concepts underlying observed knowledge components and estimate response probabilities.

3. DEEP ITEM RESPONSE THEORY

Deep-IRT combines DKVMN with IRT to retain deep knowledge tracing while estimating psychologically meaningful student ability and KC difficulty. DKVMN processes interactions through attention, prediction, and value-memory updates before the augmented networks provide IRT parameters.

  • Deep-IRT overview: Deep-IRT synthesizes DKVMN and the one-parameter IRT model to estimate student ability and KC difficulty for prediction.DKVMN processes the learning trajectory, while IRT uses the estimated parameters to predict correct responses.
  • DKVMN mechanism: DKVMN represents latent concepts in key memory and student knowledge states in value memory, using three steps: attention, prediction, and memory updating.The key memory stores concept representations, while the value memory stores evolving knowledge states.
  • Getting attention weight: At each time step, a KC embedding queries the key memory, and softmax attention weights determine how much each value-memory slot contributes.The attention vector is computed from inner products between the KC embedding and key-memory slots.
  • Making prediction: The model reads the value memory using attention, combines the read vector with the KC embedding, and predicts the probability of a correct answer.A fully connected layer with hyperbolic tangent activation forms the feature vector, followed by a sigmoid prediction.
  • Student ability and difficulty networks: The augmented networks interpret outputs as student ability and KC difficulty, while Deep-IRT retains DKVMN performance and produces difficulty estimates aligned with traditional methods.The ability and difficulty outputs are scaled through hyperbolic tangent activation and passed to the IRT response function.

4. EXPERIMENTS

The experiments evaluate Deep-IRT across five datasets using standard predictive metrics and comparisons with DKT, DKVMN, and PFA. Results indicate that Deep-IRT generally retains performance comparable to the deep learning baselines.

  • Experimental setting: Five datasets are used, comprising four public datasets and one proprietary dataset.The datasets include ASSIST2009, ASSIST2015, Statics2011, Synthetic, and FSAI-F1toF3.
  • Experimental setting: The models are trained with Adam at learning rate 0.003 and batch size 32, using gradient clipping and sequence padding for most datasets.Sequences other than Synthetic are set to length 200, with masking applied during loss computation.
  • Evaluation: Performance is reported using AUC, accuracy, and cross-entropy loss, averaged with standard deviations over five trials.AUC is interpreted so that larger values indicate better binary prediction performance.
  • Results: The DKT, DKVMN, and Deep-IRT models have similar overall performance, with most Deep-IRT–DKVMN differences not statistically significant.The authors state that the results potentially imply Deep-IRT retains DKVMN performance, while noting that large p-values do not establish equivalence.

5. DISCUSSION

The discussion examines whether Deep-IRT produces meaningful KC difficulty and student-ability estimates. Its difficulty estimates correlate strongly with most comparison sources, while its ability trajectories still show counterintuitive predictions and dependency-related weaknesses.

  • Going Deeper in Difficulty Level: Difficulty sources are compared on 131 questions from five skills in the FSAI-F1toF3 dataset using a Pearson-correlation pairs plot.The selected skills constitute around a fifth of the dataset’s interactions.
  • Going Deeper in Difficulty Level: Deep-IRT difficulty estimates align strongly with most comparison sources but correlate weakly with publisher-provided difficulty.The publisher correlation is 0.08, compared with 0.40 for item analysis and 0.39 for IRT.
  • Going Deeper in Difficulty Level: The item-analysis and IRT difficulty estimates have Pearson correlation 0.96, whereas item analysis and Deep-IRT have correlation 0.56.The discussion relates higher correlations to more similar evaluation settings and model complexity.
  • Going Deeper in Student Ability: A student trajectory visualization shows smooth prediction transitions but also counterintuitive ability and prediction changes after some responses.An incorrect response on a multi-step equation skill can coincide with increased estimated ability and prediction probability.
  • Going Deeper in Student Ability: The model does not consistently represent prerequisite relationships among skills in the examined trajectory.Correct performance on a simpler equation-solving skill is associated with decreased ability and prediction for a more advanced skill.

6. CONCLUSION

The conclusion presents Deep-IRT as an explainable deep learning knowledge tracing model that retains predictive performance while estimating KC difficulty and student ability over time. It also identifies KC dependency modeling as an area for improvement.

  • Conclusion: Deep-IRT retains deep learning knowledge tracing performance while estimating KC difficulty and student ability over time.The model combines predictive performance with interpretable estimates for students and knowledge components.
  • Conclusion: Deep-IRT difficulty estimates align with traditional methods such as IRT and item analysis.The paper presents trajectory-based difficulty estimation as a potential alternative to traditional educational testing settings.
  • Conclusion: Learning inherent KC dependencies remains an unresolved issue for Deep-IRT.The paper suggests incorporating question content into KC representations as one potential improvement.
Loading 1904.11738v1…