Source-linked AI summary
Deep Knowledge Tracing
Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, Jascha Sohl-Dickstein
TL;DR
Knowledge tracing seeks to model student knowledge over time to predict future performance, but remains difficult. This paper applies recurrent neural networks that learn latent knowledge representations from data, achieving a 25% AUC gain over the best previous benchmark result.
Problem
Knowledge tracing asks how to model student knowledge over time accurately enough to predict future interactions and personalize educational resources.
Method
Deep Knowledge Tracing applies recurrent neural networks that learn latent knowledge states and their temporal dynamics from student interaction data rather than hard-coded representations.
Results
25% AUC gain over the best previous knowledge-tracing benchmark result, with Deep Knowledge Tracing substantially outperforming previous methods across three datasets.
Takeaways & Limitations
The model supports knowledge tracing without expert annotations and enables discovering exercise influence and generating improved curricula.
Takeaways & Limitations
Binary response data limits the kinds of exercises that can be modeled.
Abstract
from arXiv · showhide
Knowledge tracing---where a machine models the knowledge of a student as they interact with coursework---is a well established problem in computer supported education. Though effectively modeling student knowledge would have high educational impact, the task has many inherent challenges. In this paper we explore the utility of using Recurrent Neural Networks (RNNs) to model student learning. The RNN family of models have important advantages over previous methods in that they do not require the explicit encoding of human domain knowledge, and can capture more complex representations of student knowledge. Using neural networks results in substantial improvements in prediction performance on a range of knowledge tracing datasets. Moreover the learned model can be used for intelligent curriculum design and allows straightforward interpretation and discovery of structure in student tasks. These results suggest a promising new line of research for knowledge tracing and an exemplary application task for RNNs.
1 Introduction
The paper introduces Deep Knowledge Tracing, using flexible recurrent neural networks to learn latent student knowledge and its temporal dynamics from interaction data. It reports a 25% AUC gain over the best previous benchmark result and shows that the model can support annotation-free structure discovery and curriculum improvement.
- Motivation: Knowledge tracing models student knowledge over time to predict future interactions and personalize content by suggesting, skipping, or delaying exercises.The task is framed around adapting learning resources to individual student needs.
- Approach: Deep Knowledge Tracing applies flexible recurrent neural networks to represent latent knowledge and temporal dynamics with learned neuron vectors.The latent representation is learned from data rather than hard-coded.
- Contributions: 25% gain in AUC over the best previous result on a knowledge tracing benchmark is reported as a main contribution.The paper presents this as an improvement over prior knowledge-tracing performance.
- Contributions: The model does not require expert annotations and can autonomously learn content substructure from exercise interactions.Expert exercise tags can be used, but they are not necessary.
- Contributions: The learned model enables discovery of exercise influence and generation of improved exercise curricula.These capabilities extend the model beyond prediction toward intelligent curriculum design.
2 Related Work
Prior knowledge-tracing work centers on BKT and related probabilistic, factor-based, ensemble, and latent-variable models, but these approaches face representational, modeling, or computational limitations. RNNs offer a flexible alternative with continuous high-dimensional latent states, motivating their use for tracing student knowledge.
- Bayesian Knowledge Tracing: BKT models student knowledge as binary concept-level latent variables updated by an HMM from correct or incorrect exercise responses.It is described as the most popular approach for temporal models of student learning.
- Limitations of Knowledge Tracing: Knowledge Tracing can use unrealistic binary understanding representations, ambiguous hidden-variable mappings, and binary response data that limit the exercises it models.Concept categories and concept-exercise mappings often require arduous, iterative Cognitive Task Analysis by domain experts.
- Alternative Models: POMDPs model open-ended learner behavior flexibly but require exponentially large state-space exploration and use discrete latent variables with hard-coded meanings.These requirements make current implementations intractable or inflexible in practice.
- Alternative Models: PFA and LFA achieve predictive power comparable to BKT, while ensembles combining these models have delivered superior results to BKT and PFA.Reported combinations include AdaBoost, Random Forest, linear regression, logistic regression, and a feed-forward neural network.
- Recurrent Neural Networks: RNNs provide flexible dynamic models with high-dimensional, continuous latent states that evolve from current inputs and prior activations.This richer representation contrasts with the dynamic HMMs used in education and supports formulating knowledge tracing as a temporal-neural-network application.
3 Deep Knowledge Tracing
The section models student exercise responses with vanilla RNN and LSTM architectures, representing interactions as fixed-length vectors and predicting correctness probabilities for subsequent exercises. It uses one-hot or random low-dimensional encodings depending on dataset size, trains with binary cross-entropy-based likelihood optimization, and applies regularization and gradient truncation.
- Modeling approach: The study applies vanilla sigmoid-unit RNNs and LSTMs to predict student responses while avoiding explicit encoding of difficult-to-quantify learning properties.The motivation is that learning depends on material, context, presentation timecourse, and individual characteristics.
- Modeling approach: Traditional RNNs encode past observations in successive hidden states that support future predictions.Inputs are mapped through hidden states to output sequences.
- Input representation: For small exercise sets, each interaction is represented as a 2M-dimensional one-hot vector combining exercise identity and correctness.Using separate representations for exercise identity and correctness degraded performance.
- Input representation: For large exercise sets, each interaction tuple receives a random Gaussian vector in R^N, with N ≪ M, to avoid impractically large one-hot encodings.The compressed-sensing motivation is that a 1-sparse one-hot signal can be exactly encoded using a fixed random Gaussian vector of length approximately log 2M.
- Prediction and training: The output vector contains predicted correctness probabilities for every problem, allowing the next response prediction to be read at the entry for the next exercise.Training minimizes negative log likelihood using stochastic gradient descent on minibatches, with dropout on readouts, truncated gradients, hidden dimensionality 200, and mini-batches of 100.
4 Educational Applications
The DKT model supports continuous assessment, adaptive curriculum design, and discovery of latent prerequisite structure. It predicts future performance from past activity and estimates how candidate exercises affect a student’s hidden knowledge state.
- Continuous assessment: DKT predicts a student’s future performance from past activity, enabling continuous assessment without formal testing when ability is tracked continuously.The paper also notes that DKT can support additional educational applications explored experimentally.
- Curriculum design: DKT can evaluate each possible next exercise by estimating the resulting hidden knowledge state for a student.After 50 answered exercises in Figure 1, the RNN can score every candidate next exercise.
- Curriculum design: Choosing an exercise sequence to maximize predicted accuracy becomes a Markov decision problem, evaluated with expectimax against mixing and blocking curricula.The curricula are tested using a particle filter with 500 particles.
- Latent structure discovery: DKT discovers latent exercise structure by assigning directed influence values Jij, and these dependencies recover the prerequisites associated with exercises.The influence characterization is based on the RNN’s correctness probability for exercise j after exercise i is answered correctly initially.
5 Datasets
The study evaluates knowledge tracing on simulated data, Khan Academy interactions, and the Assistments benchmark using AUC, with 5-fold cross-validation for non-simulated data. It compares Deep Knowledge Tracing with BKT variants and marginal-probability predictions across these datasets.
- Models are evaluated on simulated data, Khan Academy Data, and the Assistments benchmark using area under the curve (AUC).Non-simulated datasets use 5-fold cross-validation, with hyperparameters learned on training data.
- The evaluation compares Deep Knowledge Tracing with standard BKT, optimal BKT variations when available, and marginal-probability predictions.The marginal baseline predicts the probability that a student answers a particular exercise correctly.
- Simulated Data: Simulated experiments generate 2,000 students answering 50 exercises drawn from 1–5 concepts, with concept-specific latent knowledge and exercise difficulty.Each exercise is associated with a single concept and a difficulty, enabling prediction in a controlled learning setting.
- Khan Academy Data: The Khan Academy dataset contains 1.4 million exercises completed by 47,495 students across 69 exercise types in an eighth-grade Common Core curriculum.The interactions were anonymized and did not contain personal information.
- Benchmark Dataset: The Assistments 2009–2010 public benchmark is an online grade-school mathematics tutor that teaches and assesses students and is described as the largest publicly available knowledge tracing dataset.The benchmark is used to compare the model with other models.
6 Results
Deep Knowledge Tracing substantially outperformed previous methods across three datasets, matching an oracle on synthetic data and supporting curriculum design and latent-structure discovery. Its advantages were especially evident as hidden concepts increased, where BKT prediction degraded substantially.
- Prediction performance: DKT substantially outperformed previous methods across all three datasets, reaching AUC 0.85 on Khan versus 0.68 for standard BKT and 0.63 for the marginal baseline.On Assistments, DKT achieved a 25% gain over the previous best reported result, with AUC values of 0.86 and 0.69 respectively.
- Prediction performance: On synthetic data, both LSTM and RNN models predicted student responses as well as an oracle with perfect knowledge of the model parameters.Matching the oracle required modeling latent concepts, exercise difficulty, and what happened after each exercise.
- Prediction performance: BKT prediction degraded substantially as the number of hidden concepts increased because BKT cannot learn unlabelled concepts.The DKT models maintained oracle-level prediction accuracy on the synthetic dataset.
- Curriculum design: On five Assistments concepts over 30 exercises, blocking outperformed mixing, while deeper planning produced curricula with higher predicted knowledge after fewer problems.Blocking performed on par with solving expectimax one exercise deep (MDP-1), whereas looking further ahead yielded MDP-8 curricula with improved outcomes.
- Structure discovery: DKT’s conditional-influence graph perfectly clustered the five latent concepts in synthetic data, demonstrating that prediction accuracy can support latent-structure extraction.Some exercises from the same concept occurred far apart in time, yet the model recovered their clustering.
- Structure discovery: Applying the same analysis to Khan produced a graph articulating relationships among 8th-grade Common Core concepts.The analysis restricted edges to ordered exercise pairs where B followed A more than 1% of the time in the remaining sequence.
7 Discussion
The paper demonstrates that RNNs improve knowledge tracing while learning concept patterns without expert annotations and accepting any vectorized student input. It also identifies data requirements and directions for extending DKT to curriculum planning and other educational applications.
- RNNs improve prior state-of-the-art knowledge-tracing performance on the Assistments benchmark and Khan dataset.
- DKT learns concept patterns without expert annotations and can operate on any student input that can be vectorized.
- RNNs require large amounts of training data, making them better suited to online education than small classroom environments.
- Future research could add inputs such as time taken, study hint generation and dropout prediction, and model how students forget through spaced repetition.
- A controlled Khan Academy experiment will test DKT for curriculum planning by using it to propose exercises on the site.
Appendix · A LSTM Equations · B Concept Clustering
The appendix presents LSTM equations and concept-clustering analyses, including a tSNE visualization of model weights and Khan Academy exercise labels. The clustering visualization uses the best RNN model trained on synthetic data with five labeled hidden concepts.
- B Concept Clustering: The concept-clustering analysis examines whether model weights can organize concepts.The figure states that clustering concepts using model weights is difficult.
- B Concept Clustering: The analysis applies tSNE to the readout and reading weights.
- B Concept Clustering: The visualized weights come from the best RNN model.
- B Concept Clustering: The RNN model was trained on synthetic data.
- B Concept Clustering: The synthetic dataset contains five hidden concepts.
- B Concept Clustering: The five hidden concepts are labeled in the visualization.
- B Concept Clustering: A separate appendix figure shows the Khan Academy exercise labels.
C Model Insights
The model provides interpretable insights into exercise relationships, student knowledge growth, and parameter meaning. Exercise influence graphs show less structure than the relationships derived in Figure 4.
- Exercise influence graphs based on student transitions contain less structure than the relationships derived in Figure 4.
- Exercise influence graphs using empirical conditional accuracy after correct performance also contain less structure than the relationships derived in Figure 4.
- Best students show much less variance in their knowledge increase than below-average students.The comparison uses students near the 40th and 100th percentiles after 50 questions.
- The parameter bz is interpretable because its ith element generally captures the marginal probability of getting the ith exercise correct.