Source-linked AI summary
Enhancing Item Response Theory for Cognitive Diagnosis
Song Cheng, Qi Liu
TL;DR
Traditional IRT misses question-text information, does not diagnose concept proficiency, and can estimate parameters poorly for rare questions. DIRT uses deep learning with question text and student proficiency vectors to diagnose IRT parameters; experiments report that it performs best among the compared baselines.
Problem
Traditional IRT misses question-text semantics, cannot diagnose knowledge-concept proficiency, and is inaccurate for questions with only a few records.
Source: §1 · p. 1Method
DIRT combines student concept-proficiency vectors and question-text and concept embeddings with deep learning to diagnose IRT parameters, then uses the IRT formula to predict performance.
Source: §1 · p. 2Results
DIRT performs best across the reported performance-prediction metrics compared with baselines including IRT, MIRT, and DIRTNA.
Source: §4.3 · p. 5Takeaways & Limitations
Real-world experiments support DIRT’s effectiveness and interpretation power for cognitive diagnosis.
Source: §1 · p. 2Because diagnosis ground truth is unavailable, the paper validates diagnosis through performance prediction, assuming more accurate predictions generally indicate better diagnosis.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Cognitive diagnosis is a fundamental and crucial task in many educational applications, e.g., computer adaptive test and cognitive assignments. Item Response Theory (IRT) is a classical cognitive diagnosis method which can provide interpretable parameters (i.e., student latent trait, question discrimination, and difficulty) for analyzing student performance. However, traditional IRT ignores the rich information in question texts, cannot diagnose knowledge concept proficiency, and it is inaccurate to diagnose the parameters for the questions which only appear several times. To this end, in this paper, we propose a general Deep Item Response Theory (DIRT) framework to enhance traditional IRT for cognitive diagnosis by exploiting semantic representation from question texts with deep learning. In DIRT, we first use a proficiency vector to represent students' proficiency in knowledge concepts and embed question texts and knowledge concepts to dense vectors by Word2Vec. Then, we design a deep diagnosis module to diagnose parameters in traditional IRT by deep learning techniques. Finally, with the diagnosed parameters, we input them into the logistic-like formula of IRT to predict student performance. Extensive experimental results on real-world data clearly demonstrate the effectiveness and interpretation power of DIRT framework.
1 Introduction
Cognitive diagnosis supports educational applications by identifying students’ latent traits and characteristics. Traditional IRT has useful interpretable parameters, but misses question-text semantics, struggles with rare questions, and does not diagnose concept-level proficiency; DIRT is proposed to address these gaps with deep learning.
1 Introduction
- Cognitive diagnosis seeks student latent traits or characteristics and is a fundamental task in educational applications such as adaptive testing and cognitive assessment.The introduction also mentions diagnosing knowledge-concept proficiency and knowledge tracing.Source: §1 · p. 1
- Traditional IRT uses student responses but misses question-text semantics, struggles to estimate parameters for rare questions, and provides no reliable concept-level proficiency diagnosis.The paper also notes that MIRT can diagnose concept proficiency but is sensitive to concepts on which students have high proficiency.Source: §1 · p. 1
- DIRT combines a student proficiency vector and Word2Vec embeddings with deep diagnosis of IRT parameters, then predicts performance using the logistic-like IRT formula.Its deep diagnosis module uses question-text semantics to estimate student latent traits and question discrimination and difficulty, aiming to improve robustness for rare questions.Source: §1 · pp. 1–2
2 Preliminary
The preliminaries define diagnosis as estimating each student’s proficiency across knowledge concepts from student-question records. They introduce IRT and MIRT prediction parameters and describe shortcomings that motivate DIRT.
2.1 Problem Definition
- Cognitive diagnosis is framed as estimating each student’s proficiency vector across the knowledge concepts from student-question records.Each record pairs a student and question with a score; each question contains text and corresponding knowledge concepts.Source: §2.1 · p. 2
- Diagnosis is evaluated through performance prediction because ground-truth diagnosis results are unavailable, with more accurate predictions generally treated as better diagnosis.This is the paper’s stated validation rationale.Source: §2.1 · p. 2
Definition
- IRT predicts the probability of a correct response from student latent trait θ and question discrimination a and difficulty b.The text states that D is often set to 1.7 and that the theoretical ranges of a and b are [−4, 4].Source: Definition · p. 2
Shortcomings of Item Response Theory
- Traditional IRT estimates parameters from student responses alone, so it misses question-text semantics, is not robust for questions with few records, and lacks concept-level diagnosis.It diagnoses student latent trait at the question level.Source: Shortcomings of Item Response Theory · p. 2
2.3 Multidimensional Item Response Theory
- MIRT uses multidimensional student traits and concept discriminations to predict correct responses, but is sensitive to concepts where students have high latent trait.The text also states that MIRT shares IRT’s parameter-estimation shortcomings.Source: §2.3 · p. 2
3 DIRT Framework
DIRT enhances IRT with deep learning to diagnose traditional IRT parameters from student proficiency, question text, and knowledge-concept representations. Its specific implementation uses separate modules to prepare inputs, diagnose parameters, predict performance, and learn the model.
3 DIRT Framework
- DIRT combines input, deep diagnosis, and prediction modules to enhance IRT and estimate its latent trait, discrimination, and difficulty parameters.The specific implementation and its learning strategy are then described.Source: §3 · pp. 2–3, §3.1 · p. 3
Input Module
- A student proficiency vector represents mastery of each knowledge concept; question text words use Word2Vec embeddings, while associated concepts are represented by dense embeddings.The implementation obtains concept embeddings by applying a dense layer to one-hot concept vectors.Source: Input Module · p. 3
Deep Diagnosis Module
- A DNN diagnoses question-specific latent trait θ from proficiency-weighted concept embeddings, while a separate DNN estimates discrimination a from the question’s concept embeddings.The latent trait is specific to each question, and the discrimination network does not share parameters with the latent-trait network.Source: Deep Diagnosis Module · p. 3
- An attention-based LSTM estimates difficulty b by combining question text with concept relevance, which represents the concepts’ depth and width.The model uses average pooling and scales the result to match the theoretical range of b, [−4, 4].Source: Deep Diagnosis Module · pp. 3–4
Prediction Module
- The prediction module inserts the diagnosed latent trait, discrimination, and difficulty into IRT’s logistic-like formula to predict student performance.The paper states that this retains the interpretability of the diagnosed parameters as well as IRT’s performance-prediction ability.Source: Prediction Module · p. 4
DIRT Learning
- DIRT updates input- and diagnosis-module parameters by minimizing a negative log-likelihood objective with Adam.Updated parameters include student proficiency vectors, embedding weights, and the DNN and LSTM weights.Source: DIRT Learning · p. 4
4 Experiments
The experiments assess DIRT’s prediction performance, robustness on rare questions, and interpretability through a case study. They report that DIRT outperforms the compared baselines, including on rare questions, and illustrate its concept-level diagnoses.
4.1 Dataset Description
- Figure 5 reports overall student-performance prediction results on four metrics, and Figure 6 reports results for rare questions on four metrics.The experiment description identifies training splits of 60%, 70%, 80%, and 90%.Source: Figure 5 · p. 5, Figure 6 · p. 5, §4.3 · p. 5
4.2 Experimental Setup
- The evaluation compares regression metrics RMSE and MAE and classification metrics AUC and ACC against IRT, MIRT, PMF, NMF, DINA, and DIRTNA.DIRTNA uses a DNN for question texts without the attention-based LSTM; the setup also specifies 50-dimensional embeddings and a maximum question length of 30 words.Source: §4.2 · p. 5
4.3 Experimental Results
- DIRT performs best among the compared methods on student performance prediction across the tested training-data splits.The reported comparison particularly includes IRT, MIRT, and DIRTNA.Source: §4.3 · p. 5, Figure 5 · p. 5
- DIRT outperforms IRT, MIRT, and DINA on questions appearing only 1–4 times in training.The authors attribute this robustness to deep-learning representations that generalize across question texts.Source: Figure 6 · p. 5, §4.3 · p. 6
- In the case study, DIRT predicts all three questions correctly, while IRT and MIRT each miss one.The authors also report that IRT gives an abnormal difficulty value for the second question and relate MIRT’s third-question error to sensitivity to high-proficiency concepts.Source: §4.3 · p. 6, Figure 7 · p. 6
5 Related Work
The related work distinguishes continuous and discrete cognitive diagnosis models, then describes how deep learning has been applied to improve traditional methods and robustness.
5 Related Work
- Continuous cognitive diagnosis models such as IRT and MIRT represent students with one or more continuous variables, while DINA represents concept mastery with a binary indicator vector.FuzzyCDM is also cited as an example of a proposed cognitive-diagnosis model.Source: §5 · p. 6
- Prior work uses deep neural networks and LSTMs to improve traditional factorization and knowledge-tracing models, and deep learning to model heterogeneous data and robustness.The examples include DeepFM, EERNN, and MANN.Source: §5 · p. 6
6 Conclusions
The conclusion presents DIRT as a three-module framework that uses deep learning and question text to enhance IRT for cognitive diagnosis. Experiments on a real-world dataset validate its effectiveness and interpretation power.
6 Conclusions
- DIRT combines input, deep-diagnosis, and prediction modules to enhance traditional IRT using deep learning.It represents student proficiency with a vector, embeds question texts and knowledge concepts, diagnoses IRT parameters from question text, and uses those parameters in the IRT formula for prediction.Source: §6 · p. 6