Source-linked AI summary
EdNet: A Large-Scale Hierarchical Dataset in Education
Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, Jaewe Heo
TL;DR
AIEd lacks a large public benchmark that captures the diverse behaviors available in modern interactive educational systems. EdNet addresses this gap with a large hierarchical dataset collected from Santa, containing 131,441,538 interactions from 784,309 students and supporting multiple abstraction levels and AIEd tasks.
Problem
Public AIEd datasets are limited in scale and mainly capture question-solving, while modern systems provide more diverse student behaviors.
Method
The paper constructs EdNet from multi-platform Santa interaction logs and organizes student actions into four hierarchical datasets with increasing resolution.
Results
131,441,538 interactions from 784,309 students were collected, including diverse actions such as self-study, choice elimination, and course payment.
Takeaways & Limitations
EdNet provides large-scale real-world IES data for diverse AIEd tasks at multiple levels of abstraction.
Takeaways & Limitations
The question-response sequence format can omit finer-grained behaviors such as answer-choice alternation and uncertainty.
Abstract
from arXiv · showhide
With advances in Artificial Intelligence in Education (AIEd) and the ever-growing scale of Interactive Educational Systems (IESs), data-driven approach has become a common recipe for various tasks such as knowledge tracing and learning path recommendation. Unfortunately, collecting real students' interaction data is often challenging, which results in the lack of public large-scale benchmark dataset reflecting a wide variety of student behaviors in modern IESs. Although several datasets, such as ASSISTments, Junyi Academy, Synthetic and STATICS, are publicly available and widely used, they are not large enough to leverage the full potential of state-of-the-art data-driven models and limits the recorded behaviors to question-solving activities. To this end, we introduce EdNet, a large-scale hierarchical dataset of diverse student activities collected by Santa, a multi-platform self-study solution equipped with artificial intelligence tutoring system. EdNet contains 131,441,538 interactions from 784,309 students collected over more than 2 years, which is the largest among the ITS datasets released to the public so far. Unlike existing datasets, EdNet provides a wide variety of student actions ranging from question-solving to lecture consumption and item purchasing. Also, EdNet has a hierarchical structure where the student actions are divided into 4 different levels of abstractions. The features of EdNet are domain-agnostic, allowing EdNet to be extended to different domains easily. The dataset is publicly released under Creative Commons Attribution-NonCommercial 4.0 International license for research purposes. We plan to host challenges in multiple AIEd tasks with EdNet to provide a common ground for the fair comparison between different state of the art models and encourage the development of practical and effective methods.
1 Introduction
EdNet addresses the shortage of large-scale public benchmarks that capture diverse student behaviors in modern interactive educational systems. It contributes a large hierarchical interaction dataset collected from Santa for varied AIEd tasks.
- Knowledge tracing models a student’s knowledge state over time and supports tasks including learning path recommendation, score prediction, and dropout prediction.
- Existing public datasets are too small for data-driven models and mainly record question-solving activities.
- EdNet contains 131,441,538 interactions from 784,309 students collected over more than 2 years through Santa.
- EdNet includes diverse behaviors and four abstraction levels, allowing researchers to select data suited to tasks such as knowledge tracing or learning path recommendation.
2 EdNet
EdNet records student-system interactions from Santa, a multi-platform AI tutoring service for TOEIC preparation. Its records include learning materials, time spent, and fine-grained actions during study.
- EdNet contains two years of interactions from approximately 780,000 South Korean students using Santa across Android, iOS, and the Web.
- Santa responds to student actions with video lectures, response assessment, or expert commentary while preparing students for the TOEIC test.
- A recorded scenario includes purchasing a 50-day pass, solving an LC question, audio playback, and choice elimination.
- The dataset records features such as consumed learning material identity and time spent solving problems.
- 131,441,538 interactions were collected from 784,309 Santa students since 2017.
Large-scale
EdNet combines broad interaction diversity with a four-dataset hierarchical organization. Its learning-related records include activities beyond question answering, supporting analysis from multiple perspectives.
- EdNet offers the most diverse set of interactions among existing public interactive educational system datasets.
- Diversity: EdNet includes reading explanations and watching lectures, learning activities not provided in other datasets.
- Diversity: Purchasing logs may help analyze student engagement with the learning process.
- Hierarchy: The dataset is organized into KT1, KT2, KT3, and KT4, with increasing numbers and types of actions as the postfix index increases.
Hierarchy
EdNet is designed for multi-platform educational systems and provides action data at four levels of abstraction. The levels range from question responses to complete user actions for different AIEd applications.
- Hierarchy: EdNet gathers data from Santa’s iOS, Android, and Web platforms in a platform-agnostic design.
- Hierarchy: Raw student records accurately represent learning processes but require preprocessing before direct use in AIEd tasks.
- Hierarchy: KT1 through KT4 increase in resolution from question-response sequences to the complete list of user actions.
- Hierarchy: The datasets are designed for applications including knowledge tracing, score prediction, and dropout prediction.
- Hierarchy: The simplest session representation is a sequence of question-response pairs.
EdNet-KT1
EdNet-KT1 represents learning sessions as sequences of question-response pairs, providing a compact record of question-solving behavior. This format supports common knowledge-tracing models but omits important interaction details.
- EdNet-KT1 records each learning session as question-response pairs, with q_i denoting the suggested question and r_i the student’s response.The sequence is represented as (q1, r1), (q2, r2), …, (qt, rt).
- The format is used by deep-learning knowledge-tracing models including Deep Knowledge Tracing and Self-Attentive Knowledge Tracing.
- Question-response sequences concisely summarize student activities and cannot represent actions such as alternating between answer choices before submission.Modern Interactive Educational Systems can log these details, but EdNet-KT1 cannot effectively capture them in this format.
EdNet-KT2
EdNet’s hierarchical records preserve progressively finer-grained student behavior beyond question responses. KT2 captures action sequences and contexts, while higher levels incorporate learning activities and detailed actions.
- EdNet compiles students’ full action sequences, with each action representing one unit of behavior recorded in Santa’s interface.Examples include watching a video lecture, choosing a response option, and reading a passage.
- KT2 provides a more detailed representation of student behavior than question-response pairs by retaining individual actions in context.
- KT3 adds learning activities such as reading expert commentaries and watching lectures, enabling analysis of their relationship to students’ knowledge states and performance.For example, researchers can examine time spent studying commentaries and subsequent learning behavior.
- KT4 provides the complete list of actions collected by Santa to expose very fine-grained student activity.Its records support analyses tied to particular system designs, such as purchasing a paid course and subsequent studying behavior.
EdNet-KT4
KT3 and KT4 preserve increasingly detailed learning and system-interaction events. Their examples include explanations, lectures, purchases, audio controls, and answer-elimination behavior.
- KT3 records post-question learning activities, including reading an associated explanation and watching a lecture.An example follows question q790 with its explanation and lecture l540.
- KT4 records detailed actions such as purchasing an item, solving an LC question, and playing or pausing audio.The example concerns question q878 and timestamps for audio interactions.
- KT4 also captures intermediate answer behavior, including eliminating one option and selecting another.The example records eliminating ‘a’ and choosing ‘c’.
3 Related Datasets
Related datasets provide useful knowledge-tracing data but differ in scale, accessibility, structure, and recorded activities. They include synthetic responses, scaffolded tutoring interactions, course logs, and tagged mathematics questions.
- Synthetic contains 4K virtual students answering 50 questions across 1–5 concepts, with skills and item difficulties generated under Item Response Theory.
- ASSISTments combines original MCAS mathematics questions with expert-created scaffolding questions in an online tutoring and assessment system.Students first attempt the original question, with scaffolding available after failure.
- STATICS2011 contains question-solving logs from 335 engineering students in a one-semester statics course and is obtainable by request through PSLC datashop.
- Junyi Academy provides 722 mathematics questions tagged by topic and area, including 40 topic categories apart from missing-topic entries.
- PSLC datashop is an open repository containing datasets including STATICS2011, Junyi Academy, Elementary Chinese Course, and Intelligent Writing Tutor.
4 Applications of EdNet
EdNet supports applications spanning knowledge tracing, study-session dropout prediction, label-scarce pre-training, and reinforcement-learning-based student simulation for learning-path recommendation.
- 4.1 Knowledge Tracing: EdNet-KT1 enables knowledge tracing models such as SAINT to predict exercise response correctness from previous response histories.Its scale allows deep attention layers to capture complex relationships between student interactions.
- 4.2 Study Session Dropout Prediction in Mobile Learning Environments: DAS uses EdNet-KT4 to address study-session dropout prediction in mobile learning environments and outperformed existing models.A study session is defined as learning activities separated by less than one hour.
- 4.4 RL-based Learning Path Recommendation: As EdNet’s dataset level increases, simulators can offer more actions and generate more response types.The figure describes action A_t and state S_t at time step t.
- 4.3 Pre-training Tasks for Label-scarce Educational Problems: Assessment Modeling uses EdNet-KT4 for assessment-prediction pre-training and achieved state-of-the-art exam-score and review-correctness prediction.It outperformed pre-training methods from natural language processing that learn representations of learning-item contents.
- 4.4 RL-based Learning Path Recommendation: EdNet’s multiple feature levels support student simulators with different fidelity, from question responses to lecture watching, product purchasing, and answer-choice elimination.Researchers can trade off simulator simplicity against fidelity according to the goal of the trained agent.
5 Conclusion
The paper concludes that EdNet is a large, high-resolution educational dataset gathered from Santa. Its hierarchical organization supports diverse AIEd tasks at multiple abstraction levels and is intended to support continued research.
- 5 Conclusion: EdNet is a large-scale education dataset gathered by the multi-platform service Santa.The dataset records users’ activities at very high resolution.
- 5 Conclusion: EdNet is much larger than other public education datasets and contains a high-resolution record of user activities.
- 5 Conclusion: EdNet’s hierarchical structure lets researchers approach diverse AIEd tasks from different levels of abstraction.The authors state that the dataset will provide fertile soil for further AIEd developments and will remain up-to-date.