PaperlayerView exact revision on arXiv

Source-linked AI summary

Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?

Unggi Lee, Haeun Park

arXiv:2610.11135v1cs.CLcs.CYcs.LG

TL;DR

Knowledge tracing needs learner logs, which new courses and platforms may lack. This paper tests direct System-One LLM predictions and finds Jev outperforms deep KT with few or no logged learners, within the evaluated scope.

  • Problem

    Knowledge tracing predicts a learner's next-item response, but new courses and platforms may have few logged learners.

    Source: §1 · pp. 1–2

  • Method

    The study poses KT as a typed question to a frozen System-One model, adding logged-learner examples and a similar-learner statistic for JevKT.

    Source: §3 · p. 3

  • Results

    With no target-platform data, Jev reaches mean AUC .706, above the best deep KT model trained on 8 learners (.689) and Thinking-KT on all seven datasets.

    Source: §1 · p. 1

  • Takeaways & Limitations

    JevKT stays significantly ahead of deep KT through 16 logged learners and ahead on average through 64, before supervised KT catches up between 64 and 128.

    Source: §6 · p. 8

    The finding concerns one closed System-One model; Jev's training data are unknown, and contamination checks cannot rule out memorisation.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Knowledge tracing (KT) models need many logged learners, so a new course or platform starts without a usable model. In LLM-based KT the LLM generates the answer, which we call System-Two; it is either fine-tuned on the target data or reasons and votes over ten samples, which is slow and gives coarse probabilities. We ask whether an off-the-shelf System-One LLM, which returns a probability for a typed question directly in a single pass, can perform KT when few or no learners are logged. On seven datasets, Jev without any data from the target platform reaches a mean AUC of .706, above the best of 28 deep KT models trained on 8 learners (.689) and above System-Two Thinking-KT on all seven datasets (.650) at about 1/100 of its API cost. Adding examples and a similar-learner statistic from the logged learners (JevKT) raises this to .722; JevKT stays significantly ahead of deep KT up to 16 learners and ahead on average up to 64, and supervised KT catches up between 64 and 128 learners. Among the readers we tested, the gain is specific to Jev, since three other LLMs queried with the byte-identical typed request through the official System-One adapter fall below it on all seven datasets, and reader swaps and contamination checks find no evidence that the input format or memorised data explain the gain. For new learners the advantage holds from their first interactions, whereas on unseen items with all learners logged, deep KT remains ahead.

1 Introduction

Knowledge tracing needs learner logs, leaving new courses and platforms with little data for training. This study tests whether a frozen System-One LLM can predict directly from a typed question, and reports results against deep KT and System-Two LLM-KT.

1 Introduction

  • System-Two LLM-KT generates answers through reasoning or voting, using ten sampled traces and allowing only eleven probability values.The study instead asks whether a System-One model can return an answer probability directly in one pass, without generating text or reasoning traces.Source: §1 · p. 1
  • With no target-platform data, Jev0-shot reaches mean AUC .706, above the best deep KT model trained on 8 learners (.689) and Thinking-KT on all seven datasets.JevKT remains significantly ahead of the best deep KT model through 16 learners and ahead on average through 64; supervised KT catches up between 64 and 128 learners.Source: §1 · pp. 1–2
  • Controlled analyses attribute Jev's gain to the model rather than the typed format or memorised data.Other LLMs reading the same input perform worse on every dataset, while contamination checks find no sign of memorised data.Source: §1 · p. 2

2 Related Work

Prior cold-start KT work addresses limited learner logs, new learners, or transfer across platforms. LLM-based KT includes answer-generating System-Two approaches and direct-probability System-One models; this study investigates Jev.

2.1 Cold-Start KT

  • Cold-start KT covers few logged learners, new learners with short histories, and transfer across platforms.This study follows CLST's few-learner protocol and compares against the best of 28 deep KT models at each training-set size.Source: §2.1 · p. 2

2.2 System-Two and System-One LLMs for KT

  • System-Two LLM-KT generates answers through prompting or fine-tuning, while System-One models return answer probabilities directly to typed questions.Thinking-KT reads probabilities from votes across ten reasoning traces; the study evaluates commercial System-One model Jev for KT.Source: §2.2 · p. 3

3 Posing KT to a System-One LLM

The study frames knowledge tracing as predicting a learner's next-item response from interaction history. It queries frozen Jev with either the learner state alone or additional evidence from logged learners.

3 Posing KT to a System-One LLM

  • Jev is neither trained nor modified; Jev0-shot poses the learner state alone as a typed question.This tests what Jev predicts without target-platform data.Source: §3 · p. 3
  • KT predicts the probability of a correct response on the next item from the learner's prior history and the target item.In cold start, the available training pool contains N logged learners.Source: §3 · p. 3
  • JevKT adds 64 solved examples from training learners whose items share a knowledge component with the target item.It also includes the target-item first-attempt accuracy of learners most similar to the target learner.Source: §3 · p. 3
  • The selected JevKT recipe uses knowledge-component-matched examples, which help and saturate near 64 shots, plus a similar-learner statistic.Random shots do not help, and the development-split ranking transfers to test.Source: §3 · p. 3

4 Experiments

Across seven dataset cold-start tests, System-One Jev performs strongly with no logged learners; its lead narrows as logged data grow and does not extend to held-out items when all learners are available.

4.1 Experimental Setup

  • Experiments use seven public datasets with learner-level 80/20 splits and evaluate few-learner, new-learner, and held-out-item cold-start settings.Source: §4.1 · pp. 3–4

4.2 Main Results

  • With N=0, Jev0-shot reaches .706 mean AUC, beating LOKT (.671), 8-learner deep KT (.689), and Thinking-KT (.650) on all seven datasets at about 1/100 of its API cost.Table 1 reports the cold-start AUC comparison; Jev0-shot uses no data from the target platform.Source: Table 1 · p. 4, §4.2 · p. 5
  • JevKT leads the best deep KT model by +.033, +.023, and +.016 mean AUC at N=8, 16, and 32, with the average crossover between 64 and 128 learners.The lead is Holm-significant on 7/7 datasets at N=8 and 6/7 at N=16.Source: §4.2 · p. 5
  • JevKT beats the best deep KT model at every new-learner position, including interactions 2–5 (.714 vs. .691), and in 33/35 dataset–position cells.Source: §4.2 · p. 5
  • On held-out items with all learners available, the best deep KT model scores .729, ahead of JevKT (.713) and Jev0-shot (.704).The passage attributes Jev's advantage to having few learners, not to unseen items.Source: §4.2 · p. 5

5 Analysis

The analysis attributes the gain to Jev rather than the typed format, finds that logged-learner evidence adds a small benefit, and shows that System-Two reasoning does not improve the tested model. JevKT remains competitive as learner logs grow, while cost, probability reliability, and artifact checks distinguish the approaches.

5.2 What do the logged learners add?

  • At N=8, JevKT reaches .722 AUC versus .689 for the best deep KT model; its lead over deep KT is significant through 16 learners and persists on average through 64.At N=8, the prior alone reaches .706 and the similar-learner statistic alone .595; supervised KT catches up between 64 and 128 learners.Source: §5.2 · p. 6

5.1 Is it the model or the format?

  • Jev reads the identical serialized state better than Qwen3.5-9B in all 35 cells; changing the typed head or distilling Jev into Qwen weights does not reproduce the gain.On the A state, Jev reaches .761 versus .752 for the Qwen readout; with JevKT input, Jev reaches .776 versus .727 for Qwen.Source: §5.1 · p. 5, Figure 4 · p. 6

5.2 What do the logged learners add?

  • At N=16, the similar-learner statistic raises Jev0-shot from .706 to .729, while 64 KC-matched shots alone raise it to .724; together they reach .730.The statistic carries almost all the gain; shots add clearly only on DBE-KT22. On identical targets, logged evidence adds +.016 AUC to Jev, significantly on 2/7 datasets.Source: §5.2 · p. 6
  • At N=8, JevKT leads on items answered by logged learners (+.036) and items none answered (+.028), with larger leads for very low- or high-accuracy learners (+.050 and +.063).The gains are not confined to items with observed learner responses or to intermediate-accuracy learners.Source: §5.2 · pp. 6–7

5.3 Why does System-Two LLM-KT fall behind?

  • Reasoning budgets of 1024 or 2048 tokens never significantly improve Thinking-KT AUC, while Jev beats the 2048-budget version by ∆AUC +.038 to +.073.Thinking-KT probabilities pile up at 0 and 1; mean ECE is .253 for Thinking-KT and .084 for Jev0-shot.Source: Table 3 · p. 7, §5.3 · p. 7, Figure 5 · p. 7

5.4 How fast and cheap is it?

  • One JevKT prediction takes 325 ms and costs $886 per million predictions, versus $4,360 and 41.7 s for Thinking-KT; Jev0-shot costs $44 per million.DLKT is far cheaper to run, but it needs logged learners and retraining for each course.Source: §5.4 · p. 7, Table 4 · p. 8

5.5 Could the result be an artefact?

  • After validation-based hyperparameter tuning, deep KT remains below JevKT by +.070, +.042, +.030, and +.010 AUC at N=8, 16, 32, and 64, respectively.Contamination checks find no recall of public-file rows, and relabelling IDs or removing names changes Jev0-shot AUC by at most .012, never significantly.Source: §5.5 · p. 8

6 Conclusion

An off-the-shelf System-One LLM can perform KT with few or no logged learners. Jev beats deep KT trained on eight learners and prompted System-Two LLM-KT, while JevKT stays ahead until supervised KT catches up between 64 and 128 learners.

6 Conclusion

  • Without target-platform data, Jev beats deep KT trained on 8 learners on average and prompted System-Two LLM-KT on every dataset.Source: §6 · p. 8
  • With logged-learner evidence, JevKT stays ahead until supervised KT catches up between 64 and 128 learners.Source: §6 · p. 8
  • Among the readers tested, the gain comes from the model rather than the typed format; the input recipe helps other LLMs as much as Jev.Source: §6 · p. 8

Limitations

The authors bound their findings to one closed System-One model and note several limits on generalizing the comparisons or reproducing future results.

Limitations

  • The evaluation covers one closed System-One model; Jev’s training data are unknown, and contamination checks cannot rule out memorisation.Other-reader tests use LLMs behind the official adapter, a Jev-distilled student, and open-source Laya; the adapter may disadvantage those models.Source: Limitations · p. 8
  • System-Two reasoning conclusions apply to Qwen3.5-9B, and a larger reasoning model may behave differently.Source: Limitations · p. 8
  • Deep KT uses default hyper-parameters in the main comparison; broader tuning could narrow the gap, while only one Jev version is available to assess version drift.Each prediction also requires an API call, and results may change with future versions.Source: Limitations · p. 8

Ethics Statement

The study uses public, de-identified KT datasets and collects no new data from people; learner histories are sent to LLMs through an API, and predictions are framed as decision support.

Ethics Statement

  • The study uses only public, de-identified KT datasets and collects no new data from people.Source: Ethics Statement · p. 8
  • Serialized learner histories are sent to LLMs through the OpenRouter API for inference.Source: Ethics Statement · p. 8
  • KT predictions can affect what learners are shown, so outputs should support, not replace, instructors’ decisions.Source: Ethics Statement · p. 8

A Datasets and Splits

The study evaluates cold-start KT across seven varied public datasets using learner-level splits and nested training subsets, then compares Jev inputs and KT baselines under a fixed selection procedure.

A Datasets and Splits

  • The seven datasets vary in size, item and KC granularity, available text, and base rate, with interaction density affecting the evidence available to methods.Algebra2005 is an example with many interactions per item.Source: A Datasets and Splits · p. 11, Table 5 · p. 12
  • Learners with fewer than 10 interactions are excluded, and XES3G5M and Junyi are subsampled to 5,000 learners.Source: A Datasets and Splits · pp. 11–12
  • Learner-level splits use 80/20 train/test partitions; P1 and P2 score the first 50 interactions of held-out learners, while P3 holds out item IDs and scores their interactions.For P1 and P2, the first 50 interactions yield 2,000 targets per dataset; P3 scores up to 1,000 targets on held-out items.Source: A Datasets and Splits · p. 11

B Implementation Details

  • JevKT supplies a 25-interaction history, the target item, a similar-learner accuracy statistic, and 64 solved examples, then uses Jev’s typed yes/no probability without post-hoc calibration.If the context limit is exceeded, shots are halved; 3.4% of Jev targets use 32 rather than 64 shots.Source: B Implementation Details · p. 12
  • The deep KT comparison selects the best of 28 pyKT models per dataset and learner count, while LLM-KT baselines include prompted LOKT and Thinking-KT and fine-tuned CLST.Deep KT uses default hyper-parameters, and the logged API cost for all experiments was about $250.Source: B Implementation Details · p. 12

C Input-Design Exploration

  • In input exploration, similar-learner statistics provided the largest single gain, KC-matched shots helped, and random shots were neutral or slightly harmful.At N=16, the statistic alone reached .729 versus Jev0-shot’s .706, while the full JevKT input reached .730; the table reports this cold-start input ablation.Source: C Input-Design Exploration · pp. 12–13, Table 6 · p. 13

D Cold-Start Details

JevKT’s advantage over deep KT is strongest with few logged learners and fades as more are available; the new-learner advantage holds from early interactions, but not for unseen items when all learners are logged.

D Cold-Start Details

  • CLST is below JevKT in 12/12 cells with N≥8 and below the best of the eleven core deep KT models in 11/12 cells.The comparison re-implements CLST by fine-tuning Mistral-7B with LoRA on the three datasets from its original paper.Source: D Cold-Start Details · p. 13, Table 8 · p. 14
  • 7/7 datasets favor JevKT over the best deep KT model at N=8, and the average lead disappears between N=64 and 128 as the cold-start advantage shrinks.JevKT stays ahead on average through N=64; Figure 6 gives per-dataset AUC across learner counts and Figure 7 shows the difference against the best pyKT model.Source: D Cold-Start Details · p. 13, Figure 6 · p. 14, Figure 7 · p. 14
  • A synthetic-data crossover fit does not predict real-dataset crossovers (Spearman +.11, p=.82), so the paper reports observed crossovers rather than predicting them.Source: D Cold-Start Details · p. 14
  • 33/35 dataset–position cells favor JevKT for new learners at N=8, including the second-to-fifth interactions, where AUC is .714 versus .691 for the best deep KT model.The table breaks down AUC by the target’s position in the new learner’s sequence; position 1 is noisy because it has only 41–50 targets per dataset.Source: D Cold-Start Details · pp. 14–15, Table 9 · p. 15
  • With all learners available and items held out, the best deep KT model reaches .729 versus JevKT’s .713, while JevKT leads on Junyi alone.The held-out items are excluded from training, examples, and statistics; the passage concludes that unseen items alone do not favor System-One KT.Source: D Cold-Start Details · p. 15

E Model-or-Format Numbers

  • Using Jev instead of Qwen3.5-9B on the identical serialized state raises AUC in all 35 cells, while a typed head and Jev-distilled student do not exceed the plain readout.The comparison supports the authors’ conclusion that the gain comes from the System-One model rather than the typed format.Source: E Model-or-Format Numbers · p. 15

F Is the Result Specific to Jev?

The section tests whether Jev’s advantage is specific to its reader, stable over time, calibrated, and useful across settings. It also reports cost and statistical significance across cold-start sizes.

F Is the Result Specific to Jev?

  • Three other LLMs using byte-identical typed requests through the official System-One adapter score below Jev on every dataset.The adapter reads a text-verbalized probability, but a reader swap using direct token probabilities still favours Jev in 35/35 comparisons.Source: F Is the Result Specific to Jev? · pp. 15–16

G Stability of Jev’s Predictions

  • After about 51 hours, Jev’s N=0 probabilities correlate at .995–.999 with the original predictions, with mean absolute difference .008 and mean AUC change −.001.No dataset–metric pair differed significantly among 21 comparisons.Source: G Stability of Jev’s Predictions · p. 16

H Calibration

  • Jev0-shot is calibrated far better than LOKT and Thinking-KT, whose vote-frequency readouts are worst on every dataset; at N=8, JevKT and deep KT are comparable.No method is recalibrated.Source: H Calibration · p. 17

I Where Does JevKT Gain?

  • JevKT leads in every N=8 target slice with enough cases, retaining most of its lead when no logged learner answered the target item.Its lead is largest for learners with very low or very high prior accuracy.Source: I Where Does JevKT Gain? · p. 17

J Cost and Latency

  • JevKT costs about one fifth of Thinking-KT and uses one call instead of ten, while trained DLKT is cheaper per prediction but needs learners and course-specific retraining.JevKT costs more than Jev0-shot because its 64 examples lengthen the prompt.Source: J Cost and Latency · p. 17

K Significance Tests

  • At N=8, JevKT significantly beats the best deep KT model on 7/7 datasets; significance falls to 1/7 at N=256.At N=0, Jev0-shot significantly beats LOKT and Thinking-KT on all seven datasets.Source: K Significance Tests · p. 17

L Contamination Analysis

The contamination analysis combines probes of public datasets with de-identification and synthetic-data checks. None of these checks flags a dataset, and JevKT’s pattern also appears on synthetic datasets generated after the model release.

L Contamination Analysis

  • Jev and three open LLMs show no advantage from real item IDs over format-matched fake IDs, and row continuation shows no dataset-specific recall.The ID-probe comparison uses Holm correction.Source: L Contamination Analysis · p. 17
  • Relabelling IDs changes Jev0-shot AUC by at most .012 before text removal, with no significant change at the ID step; no dataset is flagged.The checks also remove names and, where available, item text.Source: L Contamination Analysis · p. 18
  • On six text-free synthetic datasets generated after Jev’s release, JevKT matches or exceeds the best deep KT model at N=8 by +.001 to +.033 AUC.Because the datasets were generated after release, Jev could not have seen them.Source: L Contamination Analysis · p. 18

M Full Per-Model Results

This section catalogs the per-model results across training-learner counts, metrics, new-learner positions, and held-out items. The supplied passages identify table coverage and evaluation setups but do not provide the table cells needed to report winners or values.

M Full Per-Model Results

Loading 2610.11135v1…