Source-linked AI summary
A Self-Attentive model for Knowledge Tracing
Shalini Pandey, George Karypis
TL;DR
Knowledge tracing must estimate student mastery from interaction histories, while existing neural approaches face challenges with sparse real-world data. SAKT uses self-attention to select relevant past exercises for predicting the next response, and the paper reports a 4.43% average AUC improvement over state-of-the-art methods. The model is also reported to be faster than RNN-based approaches.
Problem
Knowledge tracing seeks to predict students’ future exercise performance from learning histories, but RNN-based methods do not generalize well with sparse real-world data.
Method
SAKT uses self-attention to identify relevant knowledge concepts from past interactions and predict performance from those selected activities.
Results
4.43% average AUC improvement over state-of-the-art knowledge-tracing methods is reported across datasets.
Takeaways & Limitations
SAKT provides a self-attention-based knowledge-tracing approach that is reported to outperform state-of-the-art methods and run faster than RNN-based approaches.
Takeaways & Limitations
The model uses a causality layer so predictions consider only prior interactions, excluding future interaction keys.
Abstract
from arXiv · showhide
Knowledge tracing is the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities. Each student's knowledge is modeled by estimating the performance of the student on the learning activities. It is an important research area for providing a personalized learning platform to students. In recent years, methods based on Recurrent Neural Networks (RNN) such as Deep Knowledge Tracing (DKT) and Dynamic Key-Value Memory Network (DKVMN) outperformed all the traditional methods because of their ability to capture complex representation of human learning. However, these methods face the issue of not generalizing well while dealing with sparse data which is the case with real-world data as students interact with few KCs. In order to address this issue, we develop an approach that identifies the KCs from the student's past activities that are \textit{relevant} to the given KC and predicts his/her mastery based on the relatively few KCs that it picked. Since predictions are made based on relatively few past activities, it handles the data sparsity problem better than the methods based on RNN. For identifying the relevance between the KCs, we propose a self-attention based approach, Self Attentive Knowledge Tracing (SAKT). Extensive experimentation on a variety of real-world dataset shows that our model outperforms the state-of-the-art models for knowledge tracing, improving AUC by 4.43% on average.
1. INTRODUCTION
Knowledge tracing models a student’s evolving mastery from past learning interactions to predict performance on a future exercise. SAKT uses self-attention to identify relevant past knowledge concepts and reports improved accuracy and training speed.
- Knowledge Tracing: Knowledge tracing predicts whether a student will answer the next exercise correctly from past exercise interactions and response correctness.The task traces the student’s mastery level of knowledge concepts over time.
- Prior Methods: Deep Knowledge Tracing and Dynamic Key-Value Memory Network use neural architectures to model student knowledge states, but DKT has non-interpretable parameters.DKT summarizes knowledge in a hidden vector, whereas DKVMN maintains key and value matrices for exercises, concepts, and knowledge states.
- SAKT: SAKT applies a purely attention-based transformer to identify past interactions relevant to the current knowledge concept before predicting student performance.The model assigns weights to previously answered exercises while predicting performance on a particular exercise.
- Results: 4.43% average AUC improvement over state-of-the-art knowledge-tracing methods is reported across datasets.The reported improvement accompanies predictions based on relatively few relevant past activities.
- Efficiency: SAKT’s self-attention component supports parallelism, making the model an order of magnitude faster than RNN-based models.The architecture diagram presents the network used for this attention-based approach.
2. PROPOSED METHOD
SAKT models knowledge tracing with self-attention, selecting relevant prior interactions for predicting performance on the next exercise. Its architecture embeds interactions and exercises, applies causal multihead attention, then adds feed-forward, residual, normalization, and sigmoid prediction layers.
- Input and prediction: SAKT predicts the correctness of the next exercise from a student’s previous interaction sequence.Interactions encode an exercise and response, while the exercise sequence is shifted one position ahead for prediction.
- Input and prediction: Interaction tuples are encoded as integers, producing 2E possible interaction values and E possible exercise values.The encoding is yt = et + rt × E, where E is the total number of exercises.
- Embedding and position encoding: The embedding layer maps interaction and exercise sequences into latent vectors and uses padding or partitioning to support fixed-length inputs.Sequences shorter than the maximum length are left-padded, while longer sequences are partitioned into subsequences.
- Embedding and position encoding: Position encoding preserves sequence order because a student’s knowledge state evolves over time.The model separately embeds interactions and exercises before applying position information.
- Attention mechanism: Self-attention assigns relative weights to previous exercises, identifying interactions relevant to predicting the current exercise.Queries, keys, and values are linearly projected, and scaled dot-product attention determines relevance.
- Attention mechanism: Causal masking excludes future keys, ensuring prediction for interaction i uses only interactions available before it.The architecture also uses multiple attention heads and a feed-forward network to combine information and add nonlinearity.
- Output and training: Residual connections, layer normalization, and sigmoid prediction complete the architecture, with training minimizing cross-entropy loss.The sigmoid output represents the probability of a correct response to each exercise.
3. EXPERIMENTAL SETTINGS
The evaluation uses four real-world datasets and one synthetic dataset to compare SAKT with established knowledge-tracing methods. Performance is measured by AUC under a binary correctness-prediction setting using an 80%/20% train-test split.
- Datasets: The evaluation covers four real-world datasets and one synthetic dataset.The synthetic data simulate 4,000 students answering 50 exercises drawn from five concepts with varying difficulty.
- Datasets: ASSIST2009 has density 0.06, while ASSIST2015 is the sparsest dataset with density 0.05.ASSIST2015 contains 19,917 students and 708,631 interactions across 100 skills.
- Datasets: Dataset density is defined as #Unique Interactions/(#Users × #Skill tags).The dataset-statistics columns report users, skill tags, interactions, and density.
- Datasets: ASSISTChall contains 942,816 interactions from 686 students across 102 skills and has density 0.81.It is described as the richest and densest available dataset in this evaluation.
- Evaluation methodology: The binary prediction task is evaluated with Area Under Curve, comparing SAKT against DKT, DKT+, and DKVMN.Models are trained on 80% of each dataset and tested on the remaining 20%, with hidden dimensions selected from {50, 100, 150, 200}.
- Evaluation methodology: Table 3 reports the student-performance comparison across the evaluated knowledge-tracing methods.Reported results use the best hyperparameter selection for each dataset individually.
4. RESULTS AND DISCUSSION
SAKT outperforms competing knowledge-tracing methods, identifies relevant past exercises through attention, and trains substantially faster than DKT, DKT+, and DKVMN.
- SAKT achieves an AUC of 0.832 on Synthetic, exceeding DKT+ at 0.824.
- 3.16% improvement over the second-best method is reported for SAKT on ASSIST2009.
- The attention heatmap assigns zero weight to future-position pairs, reflecting the model’s use of prior interactions for each query.
- Attention weights identify relevant past exercises, including exercises far apart in the sequence but sharing a hidden concept.
- Single-head attention consistently reduces SAKT performance, while removing the self-attention block makes prediction significantly worse.
- 1.4 seconds per GPU training epoch makes SAKT 46.42 times faster than DKT+, 32 times faster than DKT, and 17.33 times faster than DKVMN.
5. CONCLUSION AND FUTURE WORK
The paper concludes that SAKT uses self-attention to model interaction history and predict performance on the next exercise from relevant past exercises. Experiments on real-world datasets show that it outperforms state-of-the-art methods and is substantially faster than RNN-based approaches.
- SAKT models student interaction history without using any RNN.
- SAKT predicts performance on the next exercise by considering relevant exercises from past interactions.
- SAKT outperforms state-of-the-art methods and is an order of magnitude faster than RNN-based approaches.