PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Tracing with Sequential Key-Value Memory Networks

Ghodai Abdelrahman, Qing Wang

arXiv:1910.13197v1cs.LGcs.AIcs.IRstat.ML

TL;DR

Knowledge tracing must model student knowledge states while handling both concept-specific mastery and long-term exercise dependencies. SKVMN combines key-value memory with recurrent Hop-LSTM sequence modeling, and it outperforms state-of-the-art KT models across five benchmark datasets.

  • Problem

    Knowledge tracing is important for educational applications, but existing models have limited ability to represent complex concept dynamics or long-term dependencies in exercise sequences.

  • Method

    SKVMN combines a key-value memory for concept states with Hop-LSTM recurrent modeling of sequential dependencies.

  • Results

    SKVMN outperformed state-of-the-art KT models on all five benchmark datasets.

  • Takeaways & Limitations

    The model can discover latent concept–question correlations and leverage exercise-sequence dependencies for improved prediction accuracy.

  • Takeaways & Limitations

    The formulation assumes questions are associated with latent concepts whose concept states describe student mastery levels.

Abstract

from arXiv · show

Can machines trace human knowledge like humans? Knowledge tracing (KT) is a fundamental task in a wide range of applications in education, such as massive open online courses (MOOCs), intelligent tutoring systems, educational games, and learning management systems. It models dynamics in a student's knowledge states in relation to different learning concepts through their interactions with learning activities. Recently, several attempts have been made to use deep learning models for tackling the KT problem. Although these deep learning models have shown promising results, they have limitations: either lack the ability to go deeper to trace how specific concepts in a knowledge state are mastered by a student, or fail to capture long-term dependencies in an exercise sequence. In this paper, we address these limitations by proposing a novel deep learning model for knowledge tracing, namely Sequential Key-Value Memory Networks (SKVMN). This model unifies the strengths of recurrent modelling capacity and memory capacity of the existing deep learning KT models for modelling student learning. We have extensively evaluated our proposed model on five benchmark datasets. The experimental results show that (1) SKVMN outperforms the state-of-the-art KT models on all datasets, (2) SKVMN can better discover the correlation between latent concepts and questions, and (3) SKVMN can trace the knowledge state of students dynamics, and a leverage sequential dependencies in an exercise sequence for improved predication accuracy.

1 INTRODUCTION

Knowledge tracing models student knowledge states from learning-activity sequences, but existing approaches struggle with complex concept dynamics and long-term exercise dependencies. SKVMN combines key-value memory with recurrent sequence modeling and achieves stronger benchmark performance while representing smoother knowledge-state evolution.

  • Motivation: Knowledge tracing models students’ knowledge states through sequences of learning activities, supporting personalized learning and educational applications.It is used in MOOCs, intelligent tutoring systems, educational games, and learning management systems.
  • Limitations of Existing Models: Traditional KT models use discrete state variables and often first-order Markov assumptions, limiting complex concept dynamics and long-term dependency modeling.These constraints can make the most recent observation represent the entire exercise history.
  • Proposed Model: SKVMN combines DKVMN’s memory capacity with DKT’s recurrent modeling, using a key-value memory and Hop-LSTM to process relevant sequential dependencies.Its triangular layer identifies dependencies so recurrent processing can hop across less relevant exercises.
  • Proposed Model: SKVMN uses a summary vector during writing to reflect the student’s current knowledge state and the prior difficulty of a new question.This addresses repeated-question cases where DKVMN would otherwise add the same knowledge growth regardless of prior answers.
  • Results: SKVMN outperformed existing KT models on five benchmark datasets and leveraged sequential dependencies for improved prediction accuracy.The evaluation also examined latent concept–question correlations and student knowledge-state dynamics.
  • Results: Compared with DKVMN, SKVMN produced smoother transitions and more progressive knowledge-state evolution across a 50-exercise ASSISTments2009 sequence.The paper presents this evolution as more accurately reflecting how a student learns.

2 PROBLEM FORMULATION

The paper formulates knowledge tracing as sequence prediction: given a student’s exercise-answer history and a new question, predict the probability of a correct answer. Latent concept states represent mastery levels and together form the student’s knowledge state.

  • Problem Definition: Knowledge tracing is formulated as a machine-learning sequence-prediction problem based on an exercise-answer history.The goal is to estimate performance at the next time step.
  • Observed Exercises: Each exercise consists of a question tag and a binary answer variable, where 0 denotes incorrect and 1 denotes correct.The question tag identifies the exercise and the answer records the observed response.
  • Prediction Target: Given history X and question qt, the model predicts pt = (yt = 1|qt, X), the probability that the student answers correctly at time t.The history contains exercises undertaken before the prediction time step.
  • Latent Knowledge State: Questions are assumed to associate with N latent concepts, whose concept states describe the student’s mastery levels.The knowledge state at time t is the set of all latent-concept states at that time.

3 SEQUENTIAL KEY-VALUE MEMORY NETWORKS

SKVMN is introduced as a knowledge-tracing model whose architecture combines memory-based representation with sequential modeling and optimization techniques for student learning sequences.

  • Model Architecture: SKVMN introduces a key-value memory for knowledge representation and Hop-LSTM recurrent modeling to leverage sequential dependencies among latent concepts.The section also discusses attention, reading, writing, and optimization procedures.

3.1 Model Overview

SKVMN uses a key-value memory to represent latent concepts and student concept states, then combines memory retrieval, recurrent sequence processing, and output prediction. After each response, the value memory is updated to reflect the latest knowledge state.

  • Memory Layer: The key matrix stores latent concepts, while the value matrix stores the student’s concept states across shared memory slots.The matrices may use different state dimensions while retaining the same number of slots.
  • Model Overview: At each time step, SKVMN retrieves the student’s knowledge state from key-value memory and predicts the probability of correctly answering the input question.The model contains embedding, memory, sequence, and output layers.
  • Memory Layer: The memory layer embeds the question, attends to relevant memory information, and reads the current knowledge state from the value matrix.Attention provides the addressing mechanism for allocating relevant information to the input question.
  • Sequence and Output Layers: The sequence layer uses recurrently connected LSTM cells whose connections follow sequential dependencies identified by a triangular layer.The output layer generates the probability of correctly answering the input question.
  • Memory Update: After observing answer yt, SKVMN updates the value matrix through a write process so it reflects the student’s latest knowledge state.The transition is from the value matrix at time t to its state at time t + 1.

3.2 Attention, Read and Write

SKVMN accesses a key-value memory to relate questions to latent concepts, read relevant student knowledge states, and update those states after each attempt.

  • Attention, Read and Write: SKVMN uses attention, read, and write processes to access and update its key-value memory.The key matrix stores latent concepts, while the value matrix stores dynamically changing concept states.
  • Attention, Read and Write: The model maps each question into an embedding and uses key-slot similarities to produce an attention vector representing question–concept correlations.The question is first represented as a one-hot vector, embedded, and compared with key slots.
  • Attention, Read and Write: The read process computes a weighted sum of value-memory slots, then combines it with the question embedding to form a summary vector.The summary vector incorporates the student’s relevant knowledge state and prior question information such as difficulty.
  • Attention, Read and Write: The write process uses the summary vector and answer correctness to calculate knowledge growth for updating the value matrix.The resulting write vector reflects both the student’s prior mastery and the outcome of attempting the question.
  • Attention, Read and Write: Erase and add gates control which information is removed from or incorporated into the student’s stored knowledge state.These gates represent forgetting and enhancement aspects of learning, respectively.

3.3 Sequence Modelling

SKVMN uses Hop-LSTM to model sequential dependencies selectively, connecting recurrent cells for exercises whose latent-concept relevance matches the current question.

  • Sequence Modelling: Long exercise histories can contain many irrelevant exercises, motivating a shorter sequence focused on exercises relevant to the current question.The ASSISTments2009 dataset has an average sequence length of 233 ± 100 questions per sequence.
  • Sequence Modelling: Attention-vector similarity is used to identify whether questions are relevant through their correlations with latent concepts.Components are mapped into low, medium, or high ranges using triangular membership functions and identity vectors.
  • Sequence Modelling: Two exercises are sequentially dependent when they share an identity vector and the earlier one is the most recent relevant exercise.This partitions the exercise history into subsequences of sequentially dependent exercises.
  • Sequence Modelling: Hop-LSTM connects cells only when their input questions are sequentially dependent, allowing it to hop across irrelevant exercises.The model therefore combines recurrent sequence modelling with relevance-based connections.
  • Sequence Modelling: For a dependent prior exercise, the current LSTM cell receives its summary vector and hidden state, updates the cell state, and produces a new hidden state.The LSTM includes forget, input, and output gates, and the hidden state is used to predict the probability of a correct answer.

3.4 Model Optimisation

SKVMN is optimized with cross-entropy loss and stochastic gradient descent, while training updates the connected model parameters and discovers latent concept states.

  • Model Optimisation: The model uses cross-entropy loss between predicted correctness probability and the true answer.The objective function is defined over the training data.
  • Model Optimisation: Memory and embedding matrices use random Gaussian initialization, while neural-layer weights and biases use Glorot uniform initialization.Glorot initialization is used for faster convergence.
  • Model Optimisation: Stochastic gradient descent optimizes the initialized parameters, with backpropagation involving only the connected Hop-LSTM cells.This parameter update follows the model’s relevance-based recurrent connections.
  • Model Optimisation: Training enables SKVMN to discover relevant latent concepts for each question and store their state values in the value matrix.The learned value matrix represents the evolving concept states used by the model.

4 EXPERIMENTS

The experiments evaluate SKVMN against established knowledge-tracing baselines across five benchmark datasets, using AUC and controlled training procedures. They also examine memory configurations and ROC performance.

  • Evaluation measure: AUC measures prediction performance, with 0.5 representing random prediction and higher scores indicating more accurate predictions.AUC is the area under the Receiver Operating Characteristic curve and ranges from 0 to 1.
  • Evaluation measure: Figure 4 compares the ROC curves of BKT, DKT, DKVMN, and SKVMN across five datasets.The datasets are ASSISTments2009, ASSISTments2015, Statics2011, Synthetic-5, and JunyiAcademy.
  • Experimental design: Datasets are split into training, validation, and testing subsets, while validation selects hyperparameters including key- and value-memory dimensions.Most datasets use 70% for training and validation and 30% for testing, followed by 5-fold cross-validation.
  • Training procedure: Training repeats five times with different initializations, and results are reported as average test AUC with standard deviation.Mini-batches contain 32 examples except for Synthetic-5, which uses 8.

5 RESULTS AND DISCUSSION

SKVMN improves prediction and concept-question clustering relative to competing models, while its sequential modeling produces smoother knowledge-state updates. Results also relate dataset difficulty and model capacity to observed performance.

  • Hyperparameters N and d: 83.63% AUC with 7.8k parameters beats 81.57% with 31k parameters on ASSISTments2009, while 82.67% with 66k beats 80.27% with 153k on JunyiAcademy.These comparisons use SKVMN and DKVMN configurations from Table 2.
  • Prediction accuracy: SKVMN outperforms BKT, DKT, and DKVMN across all five datasets.On real-world datasets, its average AUC is at least 2% higher than DKVMN; on Synthetic-5, it reaches 84.0±0.04 versus DKVMN’s 82.7±0.1.
  • Prediction accuracy: ASSISTments2015 has the lowest AUC values across models, reflecting difficulty associated with its lowest exercise-per-student ratio.The lower ratio makes training more difficult and limits effective use of sequence information.
  • Prediction accuracy: Neural models generally outperform BKT, and memory-augmented models outperform DKT in the reported AUC comparisons.The paper attributes the latter pattern to storing past learning experiences and accessing relevant information.
  • Prediction accuracy: Sequential dependencies improve SKVMN’s prediction accuracy relative to DKVMN, which primarily considers the latest observed exercise.The comparison supports using exercise-sequence information beyond the latest observation.
  • Concept-question correlation: SKVMN produces less-overlapping question clusters than DKVMN, indicating better separation of questions associated with latent concepts.Both models identify 10 latent concepts among 110 ASSISTments2009 questions.
  • Concept-question correlation: Questions grouped by SKVMN include multiple fraction-related descriptions in one cluster, providing a concrete check of discovered concept associations.Questions 13, 19, and 30 are described as Equivalent Fractions, Multiplication Fractions, and Ordering Fractions.
  • Evolution of Knowledge States: SKVMN produces smoother concept-state updates than DKVMN by considering a student’s past performance for relevant concepts.For related questions answered incorrectly and correctly, DKVMN shows sudden or significant changes while SKVMN changes more smoothly.

6 RELATED WORK

Related work progresses from Bayesian and recurrent knowledge tracing to memory-augmented models. The paper positions SKVMN as addressing limitations in concept representation and long-term sequential dependence.

  • Traditional knowledge tracing: BKT models knowledge states as binary variables estimated through Bayesian inference, but its simplified representation limits dynamics between concepts.The simplification is used to keep Bayesian inference tractable.
  • Deep knowledge tracing: DKT uses recurrent neural networks to model dynamics in past exercise sequences, but assumes a single hidden state for a student’s knowledge.The paper describes this assumption as unrealistic because knowledge can vary across learning concepts.
  • Memory-augmented tracing: DKVMN separates latent concepts in a key matrix from dynamically changing student knowledge states in a value matrix.It improved on BKT and DKT on standard knowledge-tracing benchmarks.
  • Memory-augmented tracing: DKVMN’s updates primarily use the latest exercise, which can ignore relevant past learning experience and fail to capture long-term dependencies.This limitation motivates combining memory capacity with recurrent sequence modeling.
  • Sequential key-value memory: SKVMN incorporates LSTMs into key-value memory modeling to combine recurrent sequence modeling with concept-level memory.The proposed Hop-LSTM is introduced for sequence modeling when sequences are long.

7 CONCLUSIONS

SKVMN is a knowledge-tracing model designed to overcome limitations of existing KT models by combining memory and sequential modeling. Experiments found that it outperformed state-of-the-art models across datasets.

  • SKVMN introduces a novel model for knowledge tracing.
  • Its architecture augments the memory layer with key-value memory and the sequence layer with the modified Hop-LSTM.
  • SKVMN outperformed state-of-the-art models across all evaluated datasets.
Loading 1910.13197v1…