PaperlayerView exact revision on arXiv

Source-linked AI summary

One Mastery Threshold Does Not Fit All Knowledge Tracing Models

Xianghui Meng, Yujing Zhang, Jionghao Lin

arXiv:2610.00095v1cs.LG

TL;DR

Tutoring systems use KT thresholds to determine when students stop practicing and advance, but it is unclear whether a cutoff transfers across architectures. The study evaluates threshold outcomes across models and datasets and finds that advancement patterns and balanced thresholds vary with architecture and instructional priorities.

  • Problem

    It remains unclear whether a threshold selected for one KT architecture can be applied to another without changing instructional decisions.

    Source: §1 · p. 1

  • Method

    The study evaluates threshold consequences across six KT models and four public datasets, then identifies balanced thresholds under 30 instructional settings.

    Source: §1 · p. 2

  • Results

    Across KT architectures, the same threshold produced different advancement patterns; stricter thresholds did not reliably reduce performance gaps, and stronger-prior students advanced up to 3.26 times as often.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    Thresholds should be recalibrated when the KT model or instructional objective changes and judged by observable performance, practice, advancement, and access consequences.

    Source: §6 · p. 8

    The observational logs do not support causal interpretation of post-advancement performance differences, and the analysis does not examine protected demographic groups.

    Source: §6 · p. 9

Abstract

from arXiv · show

Tutoring systems use mastery thresholds to decide when students can stop practicing and advance, but the same numerical threshold can lead to very different decisions when the underlying knowledge tracing (KT) model changes. We examine six KT models across four public educational datasets and evaluate 12 thresholds from 0.50 to 0.99 using post-advancement performance, advancement coverage, practice burden, and disparities across prior-performance groups. We also identify thresholds that balance performance, extra practice, and advancement under 30 predefined instructional settings. Bayesian Knowledge Tracing (BKT) is relatively insensitive to threshold changes, while neural models become much more selective as thresholds increase. This partly reflects different model outputs: BKT estimates latent mastery probability, whereas neural models estimate the probability of a correct next response, so the same cutoff does not represent the same level of mastery. The best-balanced threshold varied substantially across models and settings. In half of the tested settings, neural models and BKT differed by more than 0.10 in their selected thresholds, although this gap became smaller when greater priority was placed on reducing extra practice and allowing more students to advance. Stricter thresholds also did not reliably reduce performance gaps and could disproportionately restrict advancement, with stronger-prior students advancing up to 3.26 times as often as weaker-prior students. These results show that mastery thresholds should be recalibrated when the KT model or instructional priorities change and evaluated by their effects on performance, practice, advancement, and access.

1. Introduction

A mastery cutoff may not mean the same thing across knowledge-tracing architectures, yet it determines how long students practice and whether they advance. The study compares threshold consequences across models and identifies choices suited to different instructional priorities.

1. Introduction

  • BKT thresholds apply to estimated latent mastery, whereas neural KT models estimate the likelihood of a correct next response, so an identical cutoff need not represent equivalent mastery.The familiar 0.95 threshold originated in the BKT context.Source: §1 · p. 1
  • Threshold choice remains difficult to transfer across architectures because its effects on advancement decisions have largely been studied within individual tutoring systems or model families.The consequences include post-advancement performance, advancement coverage, practice burden, and disparities across prior-performance groups.Source: §1 · pp. 1–2
  • The study examines six KT models and four public datasets, then identifies thresholds balancing performance, extra practice, and advancement across 30 instructional settings.The settings assign different priorities to those outcomes.Source: §1 · p. 2
  • The same numerical threshold can produce substantially different advancement decisions across KT architectures, while stricter thresholds may disproportionately restrict advancement for students with weaker prior performance.On ASSISTments, a threshold of 0.80 yielded coverage of 0.889 under BKT and 0.113 under DKT; stronger-prior students advanced up to 3.26 times as often as weaker-prior students.Source: §1 · p. 2

2. Related Work

Prior work connects mastery rules to practice and later outcomes, but often studies thresholds within one system or model family. This paper addresses the cross-architecture decision layer and examines how instructional priorities change balanced thresholds.

2. Related Work

  • Prior studies examined mastery criteria within particular systems or model families, leaving cross-architecture transfer less clear.Pelánek and Řihák (2017) compared criteria for prediction and practice efficiency; Matayoshi et al. (2025) found a stricter rule modestly improved retention in a randomized ALEKS study, with variation across students and subjects.Source: §2 · p. 2
  • KT architectures produce different kinds of scores, so similar numerical outputs or predictive performance need not represent the same student mastery or yield the same advancement decisions.BKT estimates latent mastery; PFA accumulates prior successes and failures, while DKT, SAKT, AKT, and SAINT use next-response predictions or attention-based representations (Pavlik et al., 2009; Piech et al., 2015; Pandey & Karypis, 2019; Choi, Lee, Cho, et al., 2020; Ghosh et al., 2020).Source: §2 · p. 2
  • Fairness research has largely targeted model estimation, while this study examines the decision layer that converts model outputs into continued practice or advancement.It evaluates post-advancement performance, advancement coverage, practice burden, and disparities across prior-performance groups.Source: §2 · p. 2
  • The study also tests how the best-balanced threshold changes across 30 instructional settings that prioritize student performance, additional practice, and advancement differently..Source: §2 · p. 3

3. Method

The method calibrates thresholds to each KT model’s advancement signal and evaluates their instructional and equity consequences across four datasets. It also selects thresholds for different instructional priorities rather than assuming one cutoff fits all models.

3. Method

  • The workflow applies model-specific advancement signals across 12 thresholds, evaluates instructional and equity outcomes, and selects utility-maximizing thresholds over 30 settings.Source: §3 · p. 3, Figure 1 · p. 3

3.1 Datasets

  • Table 1 summarizes four public datasets after preprocessing and event capping; Junyi is sparse, while DBE-KT22 has 13 skills and 71.8% correctness.The datasets are chronologically ordered, with each capped at 200,000 interactions; these characteristics may limit generalization to systems with more skills, lower correctness, or sparser histories.Source: Table 1 · p. 3, §3.1 · pp. 3–4

3.2 Knowledge Tracing Models

  • All six KT models use user-level 70/10/20 train/validation/test splits; BKT models latent binary skill states, PFA uses cumulative successes and failures, DKT predicts next responses, and SAKT, AKT, and SAINT use attention-based sequence architectures.Source: §3.2 · p. 4

3.3 Model-Specific Advancement Signals

  • BKT thresholds posterior P(Know), whereas PFA and neural outputs track next-response correctness; neural signals use EWMA smoothing, and calibration does not make them mastery probabilities.Robustness tests also compare raw, smoothed, and Platt-calibrated neural signals.Source: §3.3 · p. 4

3.4 Threshold Policies and Outcome Metrics

  • Advancement begins at first threshold crossing; PMH5, coverage, and attempts to mastery are read together because stricter cutoffs can select smaller, stronger groups.Equity comparisons use Q1/Q4 PMH5 gaps and coverage ratios, with illustrative post-hoc guardrails and learner-level bootstrap intervals.Source: §3.4 · pp. 4–5

3.5 Decision Utility and Cross-Architecture Transfer

  • Utility subtracts weighted normalized attempts and uncovered sequences from PMH5, selecting a model-, dataset-, and priority-specific threshold across 30 settings.The secondary transfer analysis measures utility loss for transferred thresholds under one instructional setting; this loss depends on instructional priorities.Source: §3.5 · p. 5

4. Results

Across six KT models and four datasets, the same threshold produced different advancement decisions, and the best-balanced threshold varied with model and instructional priorities. Stricter thresholds did not reliably reduce performance gaps and could restrict advancement for weaker-prior students.

4.1 Model Prediction Quality

  • Neural KT models generally achieved higher AUC than BKT on ASSISTments, DBE-KT22, and Junyi Academy, but stronger prediction did not ensure more appropriate advancement decisions.PFA generally fell between BKT and neural models in prediction quality.Source: §4.1 · pp. 5–6

4.2 The Same Threshold Produces Different Advancement Decisions Across KT Models

  • At 0.80, BKT retained coverage above 0.57 across datasets, while DKT coverage on ASSISTments fell to 0.113 as PMH5 increased by 0.131.DKT required 2.363 attempts to advancement versus 1.419 under BKT; on EdNet KT1, DKT and AKT coverage fell below 0.01 before 0.80. DBE-KT22 was an exception, and the architecture-dependent pattern persisted across evaluation horizons.Source: §4.2 · pp. 6–7

4.3 Stricter Thresholds Do Not Reliably Reduce Prior-Performance Gaps

  • Stricter thresholds did not reliably reduce Q4-to-Q1 PMH5 gaps, while Q4-to-Q1 advancement coverage ratios reached 3.26 across evaluated settings.Guardrails satisfying tested constraints were available in 81.25% of model-dataset settings; their median utility reduction was 0.072, with a maximum of 0.221, and they do not establish causal improvements in fairness or learning.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Thresholds Depend on Both the KT Model and Instructional Priorities

  • Across 30 settings, neural-model and BKT mean selected thresholds differed by more than 0.10 in 15 settings, and neither model family consistently favored high or low cutoffs.Both selected lower thresholds when limiting extra practice and allowing more sequences to advance received greater priority; in nine settings, both selected 0.50.Source: §4.4 · p. 8
  • Under the frozen a = 0 and c = 0.05 setting, classical-to-neural transfers had mean utility loss 0.0597, versus 0.0175 for neural-to-classical transfers.These are policy-conditional sensitivity results, not a universal ranking of transfer directions.Source: §4.4 · p. 8

6. Discussion and Conclusion

The study finds that mastery thresholds translate KT outputs into different advancement patterns across models, and stricter thresholds may restrict advancement for weaker-prior students. Thresholds should therefore be evaluated as instructional decision rules, with attention to model and instructional priorities.

Discussion and Conclusion

  • The same numerical threshold can produce substantially different advancement patterns across KT models, while stricter thresholds may restrict access for weaker-prior students.Higher thresholds may raise observed post-advancement performance partly because fewer, stronger-performing sequences remain eligible to advance.Source: §6 · pp. 8–9
  • BKT thresholds latent mastery probability, whereas neural KT models use signals related to next-response correctness, so a shared cutoff need not represent the same mastery level.The authors recommend reporting observable consequences such as post-advancement performance, coverage, and additional practice, and recalibrating when models or priorities change.Source: §6 · pp. 8–9
  • The study uses observational logs, measures performance over a limited number of post-threshold responses, and does not examine protected demographic groups.Its models also exclude KT systems integrated with large language models and personalized student thresholds.Source: §6 · p. 9
Loading 2610.00095v1…