Source-linked AI summary
RKT : Relation-Aware Self-Attention for Knowledge Tracing
Shalini Pandey, Jaideep Srivastava
TL;DR
Knowledge tracing must account for how exercise relations and forgetting jointly shape the impact of past interactions, but traditional methods do not explicitly model both. RKT integrates these factors into relation-aware self-attention and outperforms state-of-the-art methods on three real-world datasets while providing interpretable attention weights.
Problem
Knowledge tracing must model students’ knowledge from interaction sequences, whose predictive impact depends on exercise relations and temporal forgetting that traditional methods do not explicitly model jointly.
Method
RKT adapts self-attention with relation coefficients derived from exercise textual content, student performance data, and a time-decaying model of forget behavior.
Results
RKT outperforms state-of-the-art knowledge tracing methods on three real-world datasets and provides interpretable attention weights.
Takeaways & Limitations
The model’s attention weights can visualize relations between interactions and temporal patterns in human learning.
Takeaways & Limitations
Future work is needed to predict relations for new exercises instead of computing them from existing data.
Abstract
from arXiv · showhide
The world has transitioned into a new phase of online learning in response to the recent Covid19 pandemic. Now more than ever, it has become paramount to push the limits of online learning in every manner to keep flourishing the education system. One crucial component of online learning is Knowledge Tracing (KT). The aim of KT is to model student's knowledge level based on their answers to a sequence of exercises referred as interactions. Students acquire their skills while solving exercises and each such interaction has a distinct impact on student ability to solve a future exercise. This \textit{impact} is characterized by 1) the relation between exercises involved in the interactions and 2) student forget behavior. Traditional studies on knowledge tracing do not explicitly model both the components jointly to estimate the impact of these interactions. In this paper, we propose a novel Relation-aware self-attention model for Knowledge Tracing (RKT). We introduce a relation-aware self-attention layer that incorporates the contextual information. This contextual information integrates both the exercise relation information through their textual content as well as student performance data and the forget behavior information through modeling an exponentially decaying kernel function. Extensive experiments on three real-world datasets, among which two new collections are released to the public, show that our model outperforms state-of-the-art knowledge tracing methods. Furthermore, the interpretable attention weights help visualize the relation between interactions and temporal patterns in the human learning process.
1 INTRODUCTION
Knowledge tracing models students’ knowledge states from exercise interactions, whose predictive impact depends on both exercise relations and temporal forgetting. RKT adapts self-attention to integrate these factors as contextual information and reports strong performance with interpretable predictions.
- Motivation: Knowledge tracing models students’ knowledge state over time to support performance prediction and personalized practice or remedial recommendations.It is difficult because student learning depends on the complexity of human cognition and knowledge acquisition.
- Research Gap: Past interactions affect predictions differently depending on their relations to the target exercise and the temporal dynamics of learning.The paper identifies exercise relation and a temporal component affecting the importance of past interactions.
- Research Gap: Existing temporal methods model elapsed time but do not jointly model relations between exercises involved in past interactions.The paper contrasts these methods with its joint treatment of exercise relations and forgetting behavior.
- Proposed Approach: RKT introduces relation-aware self-attention that uses relation coefficients derived from exercise textual content, student performance, and forget behavior.The contextual information is incorporated into self-attention to revise attention weights.
- Evaluation: Experiments on three real-world datasets report that RKT outperforms state-of-the-art algorithms while providing explanations for its predictions.The model’s attention weights are visualized to reveal relations between interactions and temporal patterns.
2 RELATED WORK
Related work spans cognitive diagnosis, probabilistic and neural knowledge tracing, exercise-relation modeling, and attention mechanisms. RKT extends self-attentive KT by incorporating exercise relations and elapsed time into attention.
- 2.1 Cognitive Diagnosis: Cognitive diagnosis models latent student mastery using one-dimensional or multi-dimensional representations of knowledge concepts.Examples include Rasch and additive factor models, alongside binary latent vectors defined using a Q-matrix.
- 2.2 Knowledge Tracing: Knowledge tracing includes Hidden Markov, factorization, tensor factorization, and recurrent-neural-network approaches for modeling evolving student knowledge.BKT uses binary concept states updated with an HMM, while DKT uses an LSTM to model knowledge evolution.
- 2.2 Knowledge Tracing: RKT extends SAKT by using exercise relations and time elapsed since the last interaction to inform self-attention.This places contextual relation and temporal information directly within the attention mechanism.
- 2.3 Relation Modeling in KT: Exercise relations can be derived from Q-matrices or textual exercise content, including predicted semantic similarity scores.Prior work commonly defines relations through shared knowledge concepts or content-based similarity.
- 2.3 Relation Modeling in KT: The paper identifies jointly using exercise-relation modeling and forget-behavior modeling in KT as an unaddressed combination.This combination motivates RKT’s relation-aware attention design.
- 2.3 Relation Modeling in KT: Attention mechanisms focus on relevant input elements for prediction and expose weights that can support model interpretation.This interpretability property motivates their use in sequence modeling and knowledge tracing.
3 PROPOSED METHOD
RKT predicts performance on the next exercise by combining exercise relations, student performance, and elapsed time within a relation-aware self-attention architecture. It infers exercise relations from textual content and performance data, incorporates forgetting behavior into relation coefficients, and uses them to modify attention weights.
- Knowledge Tracing: RKT predicts whether a student can answer the next exercise from previous interaction sequences, using exercise relations and elapsed time to determine interaction importance.Each interaction is represented by an exercise, correctness, and timestamp.
- Exercise Representation: Exercise representations are learned from textual content using word embeddings and Smooth Inverse Frequency weighting.SIF downweights unimportant words and preserves information contributing most to exercise semantics.
- Exercise-Relation Matrix Computation: RKT infers an exercise-relation matrix from student performance and textual similarity, combining Phi association and cosine similarity subject to a sparsity threshold.The matrix entry A_i,j represents the importance of performance on exercise j for performance on exercise i.
- Relation Coefficients: Relation coefficients combine exercise-relation information with forgetting behavior based on the time elapsed since each past interaction.Higher relation coefficients indicate more relevant past interactions.
- Input and Prediction Layers: RKT embeds correctness and positional information with interactions, applies a feed-forward network with ReLU, and predicts correctness probability through a sigmoid output layer.Residual connections, layer normalization, and dropout are also applied after the attention and feed-forward layers.
- Relation-Aware Self-attention Layer: The relation-aware self-attention layer adds relation coefficients to learned attention weights before computing weighted interaction representations.The addition operation incorporates contextual relations without significantly increasing computation cost, with λ controlling the combination.
4 EXPERIMENTAL SETTINGS
The experiments evaluate RKT against state-of-the-art knowledge-tracing methods using three real-world datasets, binary performance prediction, and AUC and ACC metrics. They also examine component influence and whether attention weights learn meaningful patterns.
- Research questions: The experiments ask whether RKT outperforms state-of-the-art methods, how its components influence performance, and whether attention weights learn meaningful patterns.These correspond to RQ1, RQ2, and RQ3.
- Datasets and baselines: The study evaluates RKT on three real-world datasets and compares it with state-of-the-art knowledge-tracing methods.The datasets include ASSISTment2012 and JunyiAcademy; dataset preprocessing removed students and exercises with fewer than two attempts.
- Input representation: Exercise content is represented using vocabulary words and TEX tokens, with 50-dimensional word embeddings.Mathematical formulas are transformed into TEX-code features to preserve mathematical semantics.
- Model initialization: The self-attention model uses dimension 64, maximum sequence length 50, mini-batches of 128, Adam with learning rate 0.001, and dropout 0.1.The reported setup also uses L2 weight decay of 0.00001.
- Evaluation: Student performance is evaluated as binary classification using Area Under Curve (AUC) and Accuracy (ACC).During testing, the model is updated after each received exercise response before making the next prediction.
- Datasets and baselines: The baseline comparison includes DKT, SAKT, and DKVMN knowledge-tracing methods.DKT uses an LSTM, while SAKT uses self-attention to weight previously answered exercises.
5 RESULTS AND DISCUSSION
RKT consistently outperforms the evaluated baselines by jointly modeling exercise relations, student performance, and forgetting behavior. Results also show stronger gains for sparse interaction histories, while attention visualizations expose temporal and relational patterns.
- Student Performance Prediction: RKT consistently performs better than all evaluated baselines across the reported datasets.The comparison includes methods using recurrent, memory-based, self-attention, temporal, and textual-content information.
- Limitations: The performance gain is lowest on Junyi, possibly because its relatively small exercise set allows sequential models to capture relations without explicit content-based relation learning.The paper presents this as a possible reason rather than a confirmed explanation.
- Performance Comparison with Interaction Sparsity: RKT’s performance gains are more significant for students with fewer than 10, 100, 1000, or 10000 interactions.RKT outperforms baselines in every interaction-sparsity group, indicating effectiveness with limited histories.
- Ablation Study: Removing exercise relation modeling causes the most drastic performance drop across all datasets.The ablation study also reports that encoding more information generally improves performance.
- Exercise Relation Matrix Computation: Combining student performance data and exercise textual content produces the best exercise-relation computation among the four evaluated methods.The performance-data-only method outperforms the text-only method, while the combined method outperforms both.
- Attention Weights Visualization: RKT revises attention using elapsed time and exercise relations, assigning higher weights to recent interactions and concentrating weights near related exercises.In ASSIST2012, skill-building sequences yield diagonal concentration; in POJ, weights are more distributed because students choose exercises according to their needs.
6 CONCLUSION AND FUTURE WORK
RKT predicts performance on the next exercise by combining exercise relations, student forget behavior, and self-attention over interaction history. Future work will model exercise relations directly to predict the relation of a new exercise.
- RKT predicts a student’s performance on the next exercise from interaction history using relation-aware self-attention.The model incorporates contextual information from relations with past exercises and student forget behavior.
- Exercise relations combine student performance data with textual exercise content.
- RKT models forget behavior with a time-decaying kernel function.
- Experiments on real-world datasets show that RKT outperforms state-of-the-art methods and remains interpretable through self-attention.
- Future work will model exercise relations directly rather than computing them from data, enabling prediction of a new exercise’s relation.