Source-linked AI summary
Improving Knowledge Tracing via Pre-training Question Embeddings
Yunfei Liu, Yang Yang, Xianyu Chen, Jian Shen, Haifeng Zhang, Yong Yu
TL;DR
Deep KT methods do not fully exploit question-specific information and relations among questions and skills. The paper pre-trains question embeddings with difficulty and bipartite-graph side information using PEBG, then incorporates them into deep KT models. Across three real-world datasets, PEBG improves average AUC by 8.6% and outperforms state-of-the-art models.
Problem
Deep KT models overlook question-specific information, while sparse student–question interactions make directly using questions difficult; relations among questions and skills are also underexploited.
Method
PEBG pre-trains low-dimensional question embeddings from question difficulty and explicit and implicit relations in a question–skill bipartite graph using a product layer to fuse features.
Results
8.6% average AUC improvement across three real-world datasets is reported when PEBG embeddings are used with existing deep KT models.
Takeaways & Limitations
PEBG embeddings can be incorporated into existing deep KT models and improve their performance on the evaluated datasets.
Takeaways & Limitations
The difficulty constraint includes an assumption represented by p = θ(k)θ(k)^T with θ(k) ∈ R^3.
Abstract
from arXiv · showhide
Knowledge tracing (KT) defines the task of predicting whether students can correctly answer questions based on their historical response. Although much research has been devoted to exploiting the question information, plentiful advanced information among questions and skills hasn't been well extracted, making it challenging for previous work to perform adequately. In this paper, we demonstrate that large gains on KT can be realized by pre-training embeddings for each question on abundant side information, followed by training deep KT models on the obtained embeddings. To be specific, the side information includes question difficulty and three kinds of relations contained in a bipartite graph between questions and skills. To pre-train the question embeddings, we propose to use product-based neural networks to recover the side information. As a result, adopting the pre-trained embeddings in existing deep KT models significantly outperforms state-of-the-art baselines on three common KT datasets.
1 Introduction
Knowledge tracing predicts students’ future correctness from response histories, but deep models often overlook question-specific information and face sparse question interactions. PEBG pre-trains question embeddings from difficulty and bipartite question–skill relations, and these embeddings improve existing deep KT models.
- Knowledge tracing predicts whether a student will answer the next question correctly from previous response records.
- Deep KT models commonly estimate skill mastery rather than directly modeling capability on specific questions.DKT and DKVMN are representative deep KT methods.
- Skill-level prediction misses differences among questions sharing a skill, while direct question inputs suffer from extremely sparse student–question interactions.
- PEBG pre-trains low-dimensional question embeddings from question difficulty, explicit question–skill links, implicit question similarity, and implicit skill similarity.A product layer fuses question, skill, and attribute features to preserve difficulty and relation information.
- Using PEBG embeddings in existing deep KT models outperforms state-of-the-art models, improving AUC by 8.6% on average across three real-world datasets.
2 Related Work
Prior KT research includes Bayesian, factor-analysis, and deep models, but these approaches have limitations in representing relations among skills or questions. Later work supplements skill-level modeling with question information, though practical data and relational coverage remain concerns.
- KT methods are categorized into Bayesian Knowledge Tracing, factor-analysis models, and deep KT models.
- BKT models each skill state separately and therefore cannot capture relations among skills.
- Factor-analysis models incorporate question difficulty, student ability, and response-history factors but fail to capture relations between questions.
- Deep KT models such as DKT and DKVMN use skills as network inputs, while DKVMN traces concept states with a key-value memory network.
- Some methods encode question text or estimate question difficulty, but text descriptions are difficult to acquire in practice.
3 Problem Formulation
The KT formulation represents questions and skills as nodes in a bipartite graph and defines explicit and implicit relations between them. It also incorporates question difficulty as side information for learning question embeddings.
- KT predicts the probability of correctly answering a new question from past question–correctness interactions.The target is P(c_t = 1|q_t, X).
- Questions and skills form a bipartite graph with binary adjacency matrix R, where an edge indicates that a question is associated with a skill.
- Explicit question–skill relations are determined by whether the corresponding adjacency entry r_ij equals 1.
- Question similarity links questions sharing skill neighbors, while skill similarity links skills sharing question neighbors.
- Question difficulty is represented as a vector, and attribute features may include average response time and question type.
4 Method
PEBG pre-trains question embeddings from explicit question-skill links, implicit question and skill similarities, question difficulty, and attribute interactions. A joint objective produces embeddings that can be supplied to existing deep KT models.
- Framework overview: PEBG pre-trains question embeddings using four loss functions for explicit relations, implicit similarities, and question difficulty.The four side-information targets are explicit skill-question relations, implicit question similarity, implicit skill similarity, and question difficulty.
- Input features: Question and skill vertex features are initialized randomly and updated during pre-training, while attribute features represent difficulty-related information such as response time and question type.Categorical attributes use one-hot vectors, whereas numerical attributes use scalar values.
- Graph constraints: Explicit question-skill relations are modeled by inner products between question and skill features, passed through a sigmoid and trained with cross-entropy.The sigmoid converts the estimated relation into a probability.
- Graph constraints: Implicit question and skill similarities are defined through shared graph neighborhoods and preserved by inner-product estimates with cross-entropy training.Question similarity uses shared neighboring skills, while skill similarity uses shared neighboring questions.
- Difficulty constraint: A product layer combines question, related-skill, and attribute features through pairwise inner-product interactions before transforming them into signal vectors.The related-skill feature is the average representation of skills associated with the question.
- Joint optimization: The joint optimization balances bipartite-graph constraints against the difficulty constraint, producing embeddings usable as inputs to DKT and DKVMN.The trade-off is controlled by λ, and the resulting embeddings are denoted e.
5 Experiments
The experiments evaluate PEBG-based knowledge tracing on three real-world datasets. The datasets and their statistics are summarized in Table 1.
- Experimental setup: Three real-world datasets are used to evaluate knowledge tracing models based on PEBG-pre-trained question embeddings.The evaluation section introduces these datasets before reporting model comparisons.
- Datasets: Table 1 reports the statistics of the three datasets used in the experiments.The passage identifies Table 1 as the dataset-statistics table.
- Datasets: ASSIST09 and ASSIST12 come from the ASSISTments online tutoring platform and are filtered to remove records without skills, scaffolding problems, and users with fewer than three records.After preprocessing, ASSIST09 contains 123 skills, 15,911 questions, 3,841 students, and 190,320 records; ASSIST12 contains 265 skills, 47,104 questions, 27,405 students, and 1,867,167 records.
5.2 Compared Models
The comparison includes skill-level and question-level knowledge tracing models. PEBG is evaluated by incorporating its question embeddings into existing deep KT architectures.
- Model groups: The compared models are divided into skill-level models and question-level models according to whether they use question information for prediction.Skill-level models trace student mastery of skills, whereas question-level models predict responses for individual questions.
- Skill-level models: BKT, DKT, and DKVMN are included as skill-level baselines using Bayesian, recurrent-neural-network, and key-value-memory approaches.Their inputs and internal representations differ, but all trace skill mastery.
- Question-level models: KTM is a question-level model that uses factorization machines to model interactions among student, skill, and question features.The compared question-level models use question information for question-level prediction.
- Question-level models: DKT-Q and DKVMN-Q extend DKT and DKVMN by directly using questions as inputs and predicting responses for each question.These extensions provide question-level counterparts to the original skill-level architectures.
- Question-level models: DHKT extends DKT by modeling skill-question relations while predicting student responses for individual questions.The passage describes DHKT as another question-level comparison model.
5.3 Implementation Details
Performance is evaluated with AUC under a fixed training protocol. The implementation uses specified embedding dimensions, optimization settings, regularization, and dataset splits.
- Evaluation: AUC is used as the evaluation metric for each dataset.The metric is defined as the area under the curve.
- Hyperparameters: PEBG uses 64-dimensional vertex features and 128-dimensional final question embeddings, with λ set to 0.5.The model is optimized with Adam using a learning rate of 0.001 and mini-batch size 256.
- Training protocol: Dropout with probability 0.5 is used, and each dataset is divided into 80% training and validation data and 20% test data.The training process is repeated five times according to the implementation description.
- Attribute features: Average response time and question type are used as attribute features for ASSIST09 and ASSIST12, while EdNet uses average response time.These attributes provide question-related side information for the pre-training model.
5.4 Performance Prediction
PEBG-enhanced DKT and DKVMN achieved the highest AUC across all three datasets, outperforming the corresponding original deep KT models. The results also indicate that PEBG handles sparse question interactions and complex question–skill relations more effectively than existing approaches.
- PEBG+DKT and PEBG+DKVMN achieve the highest AUC on all three datasets.
- 9.18% average AUC gain is achieved on ASSIST09 over DKT and DKVMN.PEBG+DKT and PEBG+DKVMN obtain AUCs of 0.8287 and 0.8299, compared with 0.7356 and 0.7394 for DKT and DKVMN.
- 8% average AUC increase is achieved on ASSIST12 over the corresponding DKT and DKVMN models.PEBG+DKT and PEBG+DKVMN obtain AUCs of 0.7665 and 0.7701, compared with 0.7013 for DKT and 0.6752 for DKVMN.
- 8.6% average improvement is achieved on EdNet over the original DKT and DKVMN models.
- Directly applying deep KT models to question-level prediction provides no advantage, while PEBG improves DKT and DKVMN on sparse datasets.DHKT outperforms DKT but remains below the proposed model, which the authors associate with PEBG’s use of complex question–skill relations.
5.5 Ablation Study
The ablation study evaluates explicit relations, implicit similarities, and the product layer in PEBG. Removing either relation type lowers performance similarly, while removing or replacing the product layer causes larger degradation.
- PEBG+DKT and PEBG+DKVMN perform best across the ablation settings.
- Removing explicit relations and implicit similarities causes a similar degree of performance decline.RER removes explicit question–skill relations, whereas RIS removes implicit question and skill similarities.
- Removing the product layer hurts performance badly, and replacing it with a fully connected layer also lowers performance.RPL concatenates features directly, while RPF feeds concatenated features into a fully connected layer.
- The product layer is promising for learning high-order latent patterns compared with directly concatenating features.Without the product layer, RPL and RPF use first-order and second-order neighbor information from the bipartite graph.
5.6 Embedding Comparison
The embedding visualization compares question representations learned by question-level KT models on ASSIST09. PEBG produces structured embeddings that preserve skill relations while separating questions without shared skills.
- PEBG embeddings place questions from the same skill close together and separate questions without common skills.
- DKT and DKVMN embeddings are randomly mixed, losing relations among questions and skills.
- DHKT separates different skills but fails to capture implicit similarities.
- PEBG+DKT and PEBG+DKVMN fine-tune the pre-trained embeddings for KT while retaining question–skill relations.
6 Conclusion
The paper proposes PEBG to learn low-dimensional question embeddings from question–skill bipartite graphs using a product layer. Experiments report improved deep KT performance, while visualization provides an intuitive explanation through structured embeddings.
- PEBG formulates question–skill relations as a bipartite graph and uses a product layer to learn low-dimensional question embeddings for KT.
- Experiments on real-world datasets show that PEBG significantly improves existing deep KT models.
- Visualization shows that PEBG captures question embeddings and provides an intuitive explanation of its high performance.