PaperlayerView exact revision on arXiv

Source-linked AI summary

EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction

Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, Guoping Hu

arXiv:1906.05658v1cs.CYcs.LG

TL;DR

Student performance prediction requires tracking knowledge acquisition, yet existing approaches underuse exercise materials needed for precise and interpretable analysis. The paper proposes EERNN and its explainable extension EKT, and experiments report stronger performance for the proposed models, with EKT outperforming EERNN-based models and attention outperforming Markov variants.

  • Problem

    Existing approaches mainly exploit exercising records, leaving exercise content and knowledge concepts underused for precise prediction and interpretable knowledge-acquisition analysis.

  • Method

    EERNN combines exercise records with text-based exercise encodings, while EKT extends student states to concept-specific matrices and quantifies exercise effects with a memory network.

  • Results

    All proposed EKT and EERNN models outperform baseline methods, EKT models outperform EERNN models, and attention models outperform Markov-property models.

  • Takeaways & Limitations

    The results support combining exercise materials with student records and explicitly tracking multiple concepts for prediction and interpretability.

  • Takeaways & Limitations

    EERNN’s integrated hidden vector does not explicitly explain mastery of individual knowledge concepts, motivating EKT’s matrix representation.

Abstract

from arXiv · show

For offering proactive services to students in intelligent education, one of the fundamental tasks is predicting their performance (e.g., scores) on future exercises, where it is necessary to track each student's knowledge acquisition during her exercising activities. However, existing approaches can only exploit the exercising records of students, and the problem of extracting rich information existed in the exercise's materials (e.g., knowledge concepts, exercise content) to achieve both precise predictions of student performance and interpretable analysis of knowledge acquisition remains underexplored. In this paper, we present a holistic study of student performance prediction. To directly achieve the primary goal of prediction, we first propose a general Exercise-Enhanced Recurrent Neural Network (EERNN) framework by exploring both student's records and the exercise contents. In EERNN, we simply summarize each student's state into an integrated vector and trace it with a recurrent neural network, where we design a bidirectional LSTM to learn the encoding of each exercise's content. For making predictions, we propose two implementations under EERNN with different strategies, i.e., EERNNM with Markov property and EERNNA with Attention mechanism. Then, to explicitly track student's knowledge acquisition on multiple knowledge concepts, we extend EERNN to an explainable Exercise-aware Knowledge Tracing (EKT) by incorporating the knowledge concept effects, where the student's integrated state vector is extended to a knowledge state matrix. In EKT, we further develop a memory network for quantifying how much each exercise can affect the mastery of students on concepts during the exercising process. Finally, we conduct extensive experiments on large-scale real-world data. The results demonstrate the prediction effectiveness of two frameworks as well as the superior interpretability of EKT.

1 INTRODUCTION

Student performance prediction must forecast future exercise responses while tracking knowledge acquisition, but prior approaches largely underuse exercise materials. The paper introduces EERNN and EKT to combine exercise records with content and concepts for prediction and interpretability.

  • Predicting future exercise performance supports targeted practice, learning-remedy suggestions, and personalized exercise recommendations.
  • Existing methods mainly model students’ exercise histories by representing exercises through associated knowledge concepts, underusing textual exercise information.
  • The task faces semantic exercise representation, long-term knowledge-state tracking, cold-start, and concept-level interpretability challenges.
  • EERNN combines exercise records and content, using bidirectional LSTM encodings to represent exercise semantics without expert encoding.
  • EKT extends EERNN’s integrated hidden vector into a concept-specific knowledge-state matrix and uses a memory network to quantify exercise effects on mastery.

2 RELATED WORK

Prior student-performance research spans cognitive diagnosis, knowledge tracing, matrix factorization, and deep learning, but commonly emphasizes exercise records over exercise materials. This work combines records, materials, historical attention, and explicit concept tracking to address prediction, cold-start, and interpretability.

  • Cognitive Diagnosis: Cognitive diagnosis predicts performance by inferring student states, using continuous latent traits or discrete concept-mastery vectors.
  • Knowledge Tracing: Knowledge tracing separately tracks students’ evolving knowledge states to predict performance on future exercises.
  • Matrix Factorization: Matrix-factorization methods predict unknown student scores from partially observed student–exercise performance matrices.
  • Deep Learning Methods: Deep learning methods use recurrent and memory networks to model sequential exercising, concept relationships, and prediction interpretability.
  • The paper distinguishes its approach by jointly using exercise materials, historical attention, exercise correlations for cold-start, and explicit concept-level explanations.

3 PROBLEM AND SOLUTION OVERVIEW

The paper defines student performance prediction from exercise logs and exercise materials, with two goals: tracking concept mastery and forecasting the next response. It proposes EERNN for prediction and EKT for explicit knowledge tracking.

  • Problem Definition: Each student record contains practiced exercises and binary response scores, while each exercise is represented by text content and one or more knowledge concepts.
  • Problem Definition: The problem requires tracking changes in mastery across K concepts and predicting the response score for the next candidate exercise.
  • Solution Overview: The solution overview combines all students’ exercising records with corresponding exercise materials through EERNN and EKT frameworks.
  • Solution Overview: EERNN directly predicts future performance from sequential exercising records, whereas EKT additionally tracks knowledge states explicitly.

4 EERNN: EXERCISE-ENHANCED RECURRENT NEURAL NETWORK

EERNN models student performance by combining exercise-content representations with response histories in a recurrent framework. It offers Markov and attention-based prediction strategies, while Exercise Embedding captures semantic and contextual exercise characteristics.

  • Prediction Output: EERNN provides two prediction implementations: EERNNM uses the current student state, whereas EERNNA uses attention over historical student states.EERNNM combines the current state with the next exercise representation; EERNNA replaces the current state with an attention-weighted historical representation.
  • Exercise Embedding: Exercise Embedding automatically encodes exercise text using a bidirectional LSTM and max pooling to produce a global semantic representation.Forward and backward word states capture contextual information before pooling word representations into exercise embedding xi.
  • Exercise Embedding: Exercise Embedding can capture exercise characteristics such as difficulty and distinguish individual exercises without expert encoding.The representation is learned directly from exercise text rather than manually specified concepts.
  • Modeling Process: EERNN models exercising sequences by integrating exercise embeddings with corresponding student scores in recurrent student-state representations.The framework uses recurrent updates to track states over exercising steps, with LSTM implementation supporting long-term dependencies.
  • Prediction Output: EERNNA weights historical states according to correlations between the next exercise and previous exercises, helping retain important information from long sequences.The attention mechanism is introduced because relying only on the final LSTM state may discard information through the vanish problem.
  • Model Learning: EERNN learns its parameters by minimizing the negative log likelihood of observed student exercise-response sequences.The objective uses predicted scores and actual binary scores across exercising steps.

5 EKT: EXERCISE-AWARE KNOWLEDGE TRACING

EKT extends exercise-aware recurrent modeling with concept-specific knowledge states and a memory network that quantifies exercise impacts, supporting interpretable mastery tracking and prediction.

  • Motivation: EKT addresses EERNN’s limitation that one integrated hidden vector does not explicitly explain mastery of a particular concept.The framework is designed to make concept-level knowledge acquisition interpretable.
  • EKT framework: EKT replaces EERNN’s integrated student vector with a knowledge state matrix whose vectors represent mastery of explicit concepts.The matrix evolves over time under both exercise text content and knowledge concepts.
  • Knowledge Embedding: Knowledge Embedding assigns each exercise an impact weight over concepts, accounting for correlations among concepts rather than treating them as isolated.A static memory network stores concept representations and computes these weights.
  • Student Embedding: Student Embedding uses exercise knowledge impacts to track multiple concept acquisitions simultaneously, improving interpretability.The resulting knowledge states can be associated with concepts through changes in mastery levels and exercise score records.
  • Prediction: EKT predicts performance from historical knowledge states, practiced-exercise embeddings, and the candidate exercise’s materials and concepts.The Markov implementation integrates candidate-exercise mastery with its knowledge impacts before prediction.
  • Prediction: The attention implementation extends the attentive student representation from a vector to a matrix of concept-specific states before predicting the next exercise outcome.It replaces the current knowledge-state matrix with the attentive matrix in the prediction equations.

6 APPLICATION

The application section describes using trained EERNN or EKT models to predict a student’s performance on the next exercise from her exercising history.

  • Student Performance Prediction: Both EERNN and EKT support student performance prediction as an application for proactive services such as personalized exercise recommendation.The models are applied after training to an individual student’s exercising record.
  • Student Performance Prediction: Given a student’s historical exercise, concept, and response records, the model fits her process to obtain the current state for predicting the next exercise.The application procedure applies the trained model to the student’s record before generating the next prediction.

T in EERNNM or Hp

The framework supports new students and exercises while tracking knowledge acquisition through concept-specific states and mastery estimation.

  • Cold Start: EERNN and EKT can predict for existing or entirely new students and exercises, using a non-personalized prior before adapting to new history.A new student receives an initial prediction from the trained population state, followed by personalized predictions after exercising.
  • Knowledge Acquisition Tracking: EKT estimates mastery of one concept by extracting its state from H_t and omitting the candidate exercise embedding from the prediction input.The process constructs a one-hot impact weight for the selected concept and fits the student’s record through the trained model.
  • Knowledge Acquisition Tracking: EKT estimates mastery levels using the same learned parameters with a masked exercise embedding rather than retraining the model.The masked embedding has the same dimension as the exercise input, while W3, W4, b3, and b4 remain unchanged.
  • Knowledge Acquisition Tracking: The model links learned state vectors and memory representations to concepts by comparing mastery changes with exercise score records.A state whose mastery changes align with performance on “Function” can be interpreted as the student’s knowledge state for that concept.

7 EXPERIMENTS

Experiments evaluate prediction, attention, knowledge-state tracking, and visualizations on a large mathematics dataset from an online learning system.

  • Experimental Design: The experiments assess EERNN and EKT prediction against baselines, attention mechanisms, knowledge-state tracking, and performance-prediction visualizations.These evaluation aspects are presented as the section’s four experimental goals.
  • Experimental Dataset: The mathematics dataset contains over 5 million records from 84,909 students and 15,045 exercises after filtering.Students with fewer than 10 practiced exercises and exercises with no student attempts were removed.
  • Experimental Dataset: Exercises may contain multiple concepts in a three-level expert-labeled hierarchy, including “Function” and its sub-concepts.This explicit structure supplies hierarchical knowledge labels for the experiments.
  • Experimental Dataset: Most exercises contain fewer than two concepts and features, while each concept relates to 406 exercises on average and exercise content averages about 27 tokens.These statistics motivate incorporating exercise content because concepts or features alone may lose distinctions among exercises.

7.2 Experimental Setup

The experiments configure EERNN and EKT, compare them with educational-psychology and data-mining baselines, and evaluate prediction from regression and classification perspectives.

  • Dataset and Reporting: The experimental setup also includes structural knowledge-system information and reports general-scenario results under four metrics.The dataset includes a three-level tree-based structure for knowledge concepts.
  • Exercise Embedding: The exercise embedding pipeline converts words and mathematical formulas into token sequences, then trains 50-dimensional embeddings from 1,825,767 exercises.Formula TEX features are included to preserve mathematical semantics.
  • Framework Setting: EKT tracks 37 knowledge concepts, with hidden-state dimensions set to 100 for exercise and student representations, 25 for knowledge encoding, and 50 for prediction vectors.Training uses mini-batches of 32 and dropout probability 0.1.
  • Comparison Baselines: EERNN and EKT are implemented alongside IRT, BKT, PMF, DKT, DKVMN, and LSTM-based variants for comparative evaluation.The models are tuned for fairness, with BKT implemented using open source and the others implemented in PyTorch.
  • Evaluation Metrics: Performance is assessed using MAE and RMSE for regression and ACC and AUC for classification.Lower MAE and RMSE indicate better regression results.

7.3 Student Performance Prediction

The proposed EKT and EERNN frameworks outperform the compared baselines in general prediction and remain effective for cold-start exercises, with EKT models performing best.

  • General Scenario: All proposed EKT and EERNN models outperform the baseline methods in the general student performance prediction scenario.The result is attributed to jointly using exercising records and exercise materials.
  • General Scenario: EKT models outperform EERNN models, indicating the benefit of tracking student knowledge states on multiple concepts rather than using one integrated encoding.Attention-based variants also outperform their Markov-property counterparts.
  • Cold-start Exercises: EKT, especially EKTA, performs best on cold-start exercises, followed by EERNN-based models.These models use exercise content to distinguish new exercise characteristics without retraining.
  • Cold-start Exercises: Both EKT and EERNN address cold-start prediction by representing exercises directly from their text content.Their improvement is reported as more significant for new exercises than in the general scenario.
  • Visualization Context: The visual analysis includes knowledge-mastery tracking across six concepts during 30 exercising steps.The figure compares mastery levels before and after the exercising sequence.

7.4 Effectiveness of Attention

Attention becomes increasingly useful as students’ fitting sequences lengthen, and EKT models outperform EERNN models under both ACC and AUC.

  • Attention Effects: EERNNA and EERNNM perform similarly below 40 fitting steps, but EERNNA significantly outperforms EERNNM beyond about 60 steps.Both models improve as fitting-sequence length increases.
  • Model Comparison: EKTA and EKTM outperform EERNNA and EERNNM on both ACC and AUC, respectively.The comparison uses average ACC and AUC performance.
  • Quantitative Results: The EKT-based models obtain about 0.72 on both ACC and AUC, compared with about 0.65 for EERNN-based models.These values are reported in the attention-fitting analysis.
  • Interpretation: Attention enhances selected historical student states, while the EKT framework additionally incorporates knowledge-concept information into modeling.The analysis links higher attention scores with better prediction results.

7.5 Visualizations

The visualizations show how EKT tracks concept-specific mastery and how attention connects predictions to similar prior exercises, supporting interpretation of model behavior.

  • Knowledge Acquisition Tracking: EKT represents a student’s mastery levels for explicit concepts as they change across exercising steps.The visualization selects six frequent concepts from the 37 tracked concepts for clarity.
  • Knowledge Acquisition Tracking: Correct and incorrect answers increase and decrease the modeled state of the corresponding concept, respectively.The example shows mastery of Set increasing while Function remains poorly understood after repeated failures.
  • Knowledge Acquisition Tracking: The example also contains a temporary decrease in Function mastery after a correct answer, attributed to limited records early in the process.The passage notes that tracking can improve with more exercising records.
  • Prediction Explanation: For exercise e20, both EERNNA and EKTA predict a correct answer because the student previously answered similar exercise e4 correctly.Attention scores visualize which historical exercises support the prediction.
  • Prediction Explanation: EERNNA and EKTA provide attention-based ways to analyze and explain prediction results.The paper connects this interpretability to real-world application value.

8 CONCLUSIONS

The paper develops EERNN and extends it to EKT for predicting future performance using exercise content and tracking knowledge concepts. Experiments on large-scale real-world data support the models’ effectiveness and EKT’s interpretability.

  • EERNN combines students’ exercising records with the content of corresponding exercises for performance prediction.
  • EKT extends EERNN with knowledge-concept information to track students’ knowledge states on multiple explicit concepts.
  • Markov-based EKTM and attention-based EKTA are designed under EKT, alongside corresponding EERNNM and EERNNA strategies.
  • The attention-based EKTA and EERNNA track historically focused student information and outperform the corresponding Markov-based strategies.
  • Experiments on a large-scale real-world dataset demonstrate the effectiveness and interpretability of the proposed models.
Loading 1906.05658v1…