Simulating Disengaged Students to Evaluate LLM-based Tutors

1. Introduction

• Tutors may need different responses to rapid guessing, repeated struggle, or temporary disengagement, making evaluation across learner states useful before classroom deployment.Simulated students can also support evaluation of tutoring strategies and pedagogical approaches.Source: §1 · p. 1
• Prior work describes behavioral indicators of gaming, wheel-spinning, and off-task behavior, but disengagement detectors and LLM-based student simulators are not commonly integrated in one pre-deployment protocol.Gaming can involve rapid guessing or repeated final-answer hint requests; wheel-spinning involves repeated attempts without mastery; off-task behavior can involve extended inactivity.Source: §1 · p. 2
• DAS² operationalizes five session-level states—engaged, gaming, wheel-spinning, off-task, and mixed—and examines their interpretability, simulation alignment, and relevance to tutor evaluation.The states are evaluation categories, not enduring psychological traits or diagnoses, and do not represent every form of learner engagement.Source: §1 · p. 2
• The study asks whether detector labels align with human judgments, whether simulated students reproduce intended-state behaviors, and how tutor performance varies across those states..Source: §1 · p. 2
• DAS² evaluates behavioral alignment and state controllability alongside tutor scores and rankings, including whether automated tutor evaluation aligns with human judgment.Its contributions distinguish realistic behavior from distinguishable states, and absolute tutor performance from tutor rankings.Source: §1 · p. 2

2.1 Disengagement Detection

• DAS² builds on log-based indicators of gaming, wheel-spinning, and off-task behavior to assign five session-level engagement states and validate their interpretability.Examples include rapid guessing or excessive hint use, repeated practice without mastery, and extended inactivity.Source: §2.1 · p. 2

2.2 Large Language Model Based Student Simulation

• Earlier student simulators span cognitive, rule-based, learned, and LLM-based approaches, including work on learner characteristics and knowledge levels; DAS² adds engagement states and evaluates their separability from authentic-behavior alignment.It asks both whether simulated behavior resembles authentic sessions and whether intended engagement states remain distinguishable.Source: §2.2 · p. 3

2.3 Evaluating Tutor Performance Across Student States

• Because aggregate tutor scores may obscure differences across student conditions, DAS² compares state-specific scores and tutor rankings, which can remain stable even when absolute performance changes.The cited work motivates evaluating performance across behavioral conditions and using evaluation-based rankings.Source: §2.3 · p. 3

3. DAS²: Data, Labels, and Simulators

• The six-stage DAS² workflow standardizes authentic logs, assigns literature-grounded student states, configures simulators, evaluates behavioral alignment and state separability, and assesses AI tutors.ASSISTments09 is the primary dataset; ASSISTments12, Eedi, and EdNet-KT1 serve distinct related-domain, cross-dataset, and response-time/off-task roles.Source: §3 · p. 3

3.1 Datasets and Detectable Learner States

• ASSISTments09 supports simulator training and human label validation, ASSISTments12 related-domain evaluation, Eedi cross-dataset evaluation, and EdNet-KT1 response-time and off-task analyses.EdNet-KT1 lacks correctness information needed to identify gaming and wheel-spinning.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Rules for Labeling Learner States

• Literature-grounded rules label gaming, wheel-spinning, and off-task behavior using response speed, hints, repeated attempts, correctness, and inactivity; other sessions are engaged or mixed.The rules include more than 10 attempts on one knowledge component without three consecutive correct responses, or an inactivity gap longer than five but shorter than 30 minutes.Source: §3.2 · pp. 4–5

3.3 Session-Level Personas

• Each simulator receives a persona instructing it to generate behavior for one of five states identified in authentic logs; the state is a behavioral category, not a student trait..Source: §3.3 · p. 5

3.4 Human Validation of Student-State Labels

• Two coders independently labeled 100 ASSISTments09 sessions from anonymized summaries, with 25 sampled from each of four DAS²-assigned groups.They were blind to DAS² labels and each other's annotations; agreement calculations retained all sessions, while human-to-DAS² comparisons used the 84 cases both coders labeled identically.Source: §3.4 · p. 5

3.5 Student Simulator Configurations

• V1 is prompt-only GPT-4o, V2 adds three similar labeled sessions as examples, and V3 is LoRA-fine-tuned Qwen2.5-7B; V3-cal post-processes correctness rates.V2 is exploratory because its retrieval pool does not exclude held-out evaluation sessions; V3-cal changes correctness while leaving timing, hints, interaction order, and persona unchanged.Source: §3.5 · pp. 5–6

4.1 Behavioral Alignment Metrics

• W1, KS D, MMD, and JS divergence measure simulated-versus-authentic behavioral alignment, with lower values indicating closer alignment.W1 and KS D compare log-transformed response-time distributions, MMD adds another response-time comparison, and JS compares joint correctness, hint-use, and response-time distributions.Source: §4.1 · p. 6

4.2 Student-State Representation and Separability

• A V3 persona ablation compares intended-state instructions with an engaged-state instruction, while silhouette scores and five-state classification accuracy assess whether generated states remain distinguishable.The separability analysis compares V1, V3-persona, and V3-cal; V2 is excluded because its retrieval pool does not ensure separation from evaluation data, and five-state chance accuracy is 20%.Source: §4.2 · p. 6

4.3 AI Tutor Evaluation Across Student States

• Five tutors are evaluated with simulated students in five states, primarily over 10 turns and independently over 20 turns, using ratings for relevance, scaffolding, and engagement recovery.The protocol compares each tutor's engaged and disengaged scores, engaged-versus-disengaged rankings, automated ratings with human ratings, and results under a second automated judge.Source: §4.3 · pp. 6–7

5.1 Human Validation of Student-State Labels

• Coders agreed on 84% of 100 ASSISTments09 sessions, and human labels matched DAS² labels in 81% of the 84 agreed cases.Gaming was especially difficult to distinguish from wheel-spinning; its precision was 0.43.Source: §5.1 · p. 7

5.2 Behavioral Alignment and Student-State Distinguishability

• Persona conditioning reduced the gaming correctness gap from 0.54 to 0.20 and the wheel-spinning correctness gap from 0.51 to 0.18.It also reduced the gaming-rate gap from 0.51 to 0.30; post-hoc correctness calibration changes the correctness marginal without changing response timing.Source: §5.2 · p. 7, Table 2 · p. 8
• Five-state classification accuracy was 76.2% for V1 and 30.4% for V3, while V3-cal reached 45.2% but still showed substantial state overlap.V3's accuracy was only moderately above the 20% chance level, and its near-zero silhouette score indicated overlap.Source: §5.2 · p. 8
• On ASSISTments09, fine-tuned Qwen2.5-7B reduced response-time W1 from 1.00 to 0.50 versus prompt-only GPT-4o, while the values remained domain-dependent across datasets.KS, MMD, and JS also decreased from 0.60 to 0.32, 0.31 to 0.10, and 0.62 to 0.30, respectively.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Tutor Performance Across Student States

• Tutor rankings were identical across 10- and 20-turn evaluations (Spearman ρ = 1.00), although state-specific scores changed by approximately 0.01 to 0.03.The 10-turn grid was primary, and the 20-turn grid was a robustness check.Source: Table 3 · p. 8, §5.3 · p. 8
• Overall judge-human correlation reached 0.82, below the predefined 0.85 benchmark, and automated tutor evaluation was not sufficiently aligned with human judgment to be fully validated.Revising the relevance criterion raised its correlation with human judgments from 0.33 to 0.61.Source: §5.3 · pp. 8–9

• Tutor scores varied across student states, but tutor rankings remained comparatively stable; automated evaluator judgments also aligned incompletely with human judgments.Source: §6 · p. 9
• Closer alignment with authentic interaction distributions did not necessarily produce more distinguishable student states.A simulator may reproduce realistic response-time distributions while generating gaming, wheel-spinning, and off-task sessions that are difficult to distinguish from one another.Source: §6 · p. 9
• Stable tutor rankings can coexist with different responses to disengaged students, so state-specific performance reveals information an overall score or ranking misses.Source: §6 · p. 9
• Human validation covered 100 ASSISTments09 sessions, with limited off-task cases and ambiguity between gaming and wheel-spinning.The operational rules capture selected forms of disengagement, tutor evaluation covered five models and relied mainly on one automated judge, and simulated interactions do not measure actual student learning.Source: §6 · p. 9