Source-linked AI summary
Simulated Students in Tutoring Dialogues: Substance or Illusion?
Alexander Scarlatos, Jaewook Lee, Simon Woodhead, Andrew Lan
TL;DR
Simulated students could reduce reliance on costly real-student evaluation, but their realism is not well measured and prompting alone is unreliable. This paper defines reference-based behavioral, cognitive, and linguistic metrics and benchmarks prompting, supervised fine-tuning, and reinforcement learning on real math tutoring dialogues. Fine-tuning and reinforcement learning outperform prompting, yet LLMs remain far from fully resembling real student behavior.
Problem
Existing simulated-student research lacks sufficient turn-level metrics for measuring realistic behavior, while prompting often fails to reliably simulate students.
Method
The paper defines six behavioral, cognitive, and linguistic dimensions with reference-based metrics, then benchmarks prompting, supervised fine-tuning, and reinforcement learning.
Results
Fine-tuning and reinforcement learning outperform prompting, but LLMs remain far from fully resembling real student behavior in dialogues.
Takeaways & Limitations
The results motivate further work on more realistic simulated students and stronger evaluation of their behavior.
Takeaways & Limitations
Experiments use a single math dialogue dataset, and the reference-based metrics require existing tutor–student dialogues.
Abstract
from arXiv · showhide
Advances in large language models (LLMs) enable many new innovations in education. However, evaluating the effectiveness of new technology requires real students, which is time-consuming and hard to scale up. Therefore, many recent works on LLM-powered tutoring solutions have used simulated students for both training and evaluation, often via simple prompting. Surprisingly, little work has been done to ensure or even measure the quality of simulated students. In this work, we formally define the student simulation task, propose a set of evaluation metrics that span linguistic, behavioral, and cognitive aspects, and benchmark a wide range of student simulation methods on these metrics. We experiment on a real-world math tutoring dialogue dataset, where both automated and human evaluation results show that prompting strategies for student simulation perform poorly; supervised fine-tuning and preference optimization yield much better but still limited performance, motivating future work on this challenging task.
1 Introduction
LLM-based tutoring increasingly relies on simulated students because real-student interaction is difficult to scale, yet realistic simulation remains poorly measured and difficult to achieve. The paper addresses this gap by defining evaluation dimensions and benchmarking simulation methods.
- LLM tutoring research uses simulated students because training with real students is difficult to scale.Existing alternatives include expert role-play and prompting LLMs to simulate students.
- Prompted LLMs often fail to reproduce realistic student behavior, including cognitive characteristics such as learning patterns.The paper also notes limited turn-level evaluation metrics for realistic student simulation.
- The paper identifies six dimensions of student behavior: dialogue acts, correctness, error-making, knowledge acquisition, language use, and tutors’ responses.These dimensions support reference-based evaluation of simulated responses against ground-truth responses.
- The study benchmarks prompting, supervised fine-tuning, and multi-objective reinforcement-learning methods on 2,000 real tutor–student dialogues.Automated and human evaluations indicate that prompting often misses most behavior dimensions, while fine-tuning and reinforcement learning improve alignment.
2 Methodology
The methodology models student turns conditionally on dialogue context and evaluates their realism across behavioral, cognitive, linguistic, and dialogue-progression dimensions. It combines reference-based metrics with language-model classifiers and tutor-response likelihoods, while noting risks from metric-based reinforcement learning.
- LLM-based Student Simulation: A student simulator generates each predicted utterance from prior student turns, tutor turns, the question, and an optional prompt containing student context.The simulator may be pretrained, supervised-fine-tuned on student utterances, or further refined with reinforcement learning.
- Evaluating Simulated Student Turns: The evaluation suite measures dialogue acts, correctness, errors, knowledge acquisition, linguistic similarity, and the likelihood of inducing the tutor’s actual response.These metrics compare simulated turns with reference student turns or assess their fit within the dialogue progression.
- Evaluating Simulated Student Turns: Dialogue-act similarity is binary, while correctness and error similarity are evaluated only when the reference turn supports those comparisons.Dialogue acts are classified with a fine-tuned local model; correctness and error judgments use GPT-5 mini, and errors are compared for equivalence.
- Evaluating Simulated Student Turns: Knowledge-acquisition similarity compares five equal-sized quantile bins of mastery changes rather than raw knowledge-tracing deltas.The metric uses estimated mastery changes for relevant knowledge components and an inverted average distance between delta quantiles.
- Evaluating Simulated Student Turns: The tutor-response metric estimates whether the ground-truth next tutor turn is likely given the simulated student utterance.It treats the simulated response as fitting the dialogue when it induces the tutor response associated with the reference turn.
- Tuning with RL: The reinforcement-learning pipeline may reward hacking because models can overfit errors in LLM-based evaluations, motivating expert human evaluation when such metrics guide training.This caution concerns the reliability of metrics used as training rewards.
3 Experiments
The experiments benchmark prompting and fine-tuning approaches on real math tutoring dialogues and validate automated metrics with human judgments. Prompting performs well mainly on correctness, whereas fine-tuned methods perform better across several other dimensions, with error prediction remaining difficult.
- Experimental Setup: The study benchmarks prompting and fine-tuning methods on a real-world dataset of math tutor–student dialogues.The dataset contains middle-school students solving multiple-choice problems with tutor guidance, and the models include Llama-3.1-8B-Instruct and Llama-3.2-3B-Instruct.
- Student Simulation Methods: Prompting methods include zero-shot, OCEAN persona, oracle summaries, in-context learning, and reasoning-based prompting.These methods vary the information and reasoning supplied to the language model when generating student turns.
- Human Evaluation: Human evaluation covers 190 turns across 38 dialogues, with expert evaluators judging ground-truth and simulated student turns.Evaluators label dialogue acts and correctness after reviewing five consecutive student turns per dialogue.
- Quantitative Results: Prompting-based methods perform well on correctness, while fine-tuned methods perform well on acts, knowledge, linguistic similarity, and tutor responses.All methods except Oracle perform poorly on error prediction.
4 Results
Fine-tuned simulated students generally outperform prompting methods across automated and human evaluations, although prompting can perform well on correctness by favoring the majority class. Results also show that fine-tuning improves nuanced behavior but does not fully capture student diversity, and metric-targeted rewards can trade off performance across dimensions.
- 4.1 Quantitative Results: Fine-tuning methods generally outperform prompting methods on Acts, Knowledge, Cos. Sim., ROUGE-L, Tutor Resp., and usually Errors.Oracle is an exception on Errors because its prompt leaks exact student errors.
- 4.1 Quantitative Results: Prompting methods perform better on Correctness mainly by generating correct responses, the majority class, rather than by reproducing broader student behavior.The authors interpret this pattern as evidence that prompting anticipates high-level behavior but misses more nuanced details.
- 4.1 Quantitative Results: Reasoning improves several prompting metrics, but even Oracle cannot outperform much smaller fine-tuned models on most metrics despite its leaked-information cheat sheet.OCEAN is stronger on Acts, ICL on linguistic metrics, and Zero-Shot performs worst among prompting methods.
- 4.1 Quantitative Results: DPO only slightly outperforms SFT across metrics and is slightly worse on Errors, possibly because SFT-generated candidates provide too few positive error examples for RL.The larger 8B model also beats the 3B model by small, consistent margins except on Errors.
- 4.2 Human Evaluation: Human evaluation broadly matches automated results, while automated metrics show very high agreement with human-assigned scores across labels.Errors have lower agreement because they are measured only on incorrect turns and are highly imbalanced.
- 4.3 Qualitative Analysis: Fine-tuned models more closely match real-student language but produce very short, relatively uniform responses and rarely reproduce typos, excessive punctuation, or individual behavioral variation.DPO 8B averages 2.28 words per response, compared with 10.89 for ICL and 4.11 for real students.
- 4.4 Ablation Study: Metric-specific DPO rewards usually improve the targeted metric but can substantially alter unrelated metrics, revealing trade-offs among aspects of student behavior.Training on Correctness yields high Correctness but the lowest performance on most other metrics, whereas Knowledge produces the highest Errors, Cos. Sim., and Tutor Resp.
5 Conclusions and Future Work
The paper introduces student-utterance simulation and reference-based realism metrics, then benchmarks prompting and fine-tuned methods. Fine-tuning and reinforcement learning outperform prompting, but simulated students still fall short of fully resembling real student behavior.
- The paper introduces student utterance simulation in tutoring dialogues and a suite of reference-based metrics for measuring realism against reference student turns.
- The benchmark compares a wide range of fine-tuned and prompting-based student simulation methods.
- Fine-tuning and RL outperform prompting, but LLMs remain far from fully resembling real student behavior in dialogues.
- Future work should improve training methods, incorporate prior student information, extend evaluation from turns to dialogues, develop reference-free metrics, and test other domains.
Limitations
The study is constrained by its single math-dialogue dataset, reference-based metrics, incomplete human evaluation, limited affect measurement, proprietary metric models, and only slight RL gains over SFT.
- Experiments use a single math dialogue dataset, so generalization to other datasets or domains remains unestablished.
- Reference-based metrics limit evaluation to settings with existing tutor-student dialogues, and two metrics lack human-evaluation agreement.
- The study does not measure affect or emotional state, which may matter in other domains.
- The annotations and Correctness and Errors metrics rely on proprietary LLMs because smaller open-source models were less reliable in preliminary experiments.
- RL performs only slightly better than SFT, leaving the explanation for this result to future work with more advanced RL methods.
Ethical Considerations
Simulated students could support low-stakes practice, safer tutor development, and faster evaluation, but biased or unrepresentative simulations could harm underrepresented students and reduce educational quality.
- Simulated students could support low-stakes human-tutor practice, A/B testing that reduces risks to real students, and AI-tutor training.
- Reliable evaluations could accelerate simulated-student research and help avoid deploying unreliable models in real educational settings.
- Training-data or model bias may make simulations less representative of underrepresented groups and potentially cause unintended harm.
- Over-reliance on unrepresentative simulated students could reduce educational-tool quality and harm learning outcomes.
- The authors recommend demographic-group bias evaluation and thorough A/B testing with real students before widespread deployment.
B.2 Additional Metric Details
The metric framework defines dialogue acts for one-on-one math tutoring and uses several learned or prompted components to evaluate simulated student behavior.
- Dialogue acts were refined from prior definitions to fit one-on-one math tutoring while balancing low ambiguity and behavioral descriptiveness.
- The act taxonomy removed an Other category to improve labeling accuracy and collapsed correct and incorrect answer acts into the correctness metric.
- Acts and correctness models use supervised fine-tuning with label-text outputs and greedy decoding at inference.
- The knowledge-tracing model is conditioned on estimated OCEAN personas, increasing AUC from 0.5940 to 0.6557 without leaking test-time information to the student model.
- Evaluation uses GPT-4.1 for several ground-truth labels and standard software libraries for embeddings, fine-tuning, DPO, inference, and ROUGE-L.
C Additional Dataset Details
The dataset is anonymized UK tutoring dialogue data with documented dialogue, turn, correctness, and subject characteristics. Additional materials define training statistics and the dialogue-act label set used in analysis.
- The anonymized dataset contains only students from the United Kingdom and provides no demographic information about individual students or tutors.
- Dialogues average 23.42 turns, with tutors initiating 82.63% of them; student and tutor turns average 4.11 and 14.84 words, respectively.
- GPT-4.1-generated answers, solutions, and option explanations were used to identify unsolvable questions, removing 60 dialogues, or 3% of the dataset.
- Table 5 reports fine-tuned-model validation performance and training/testing runtimes, using accuracy for Acts and Correctness and AUC for Knowledge.
- The dialogue-act inventory defines the possible student acts at any turn in a dialogue.
- Each dialogue is associated with granular subject-defined knowledge components, plus a Default component for turns outside the listed subjects.
D.1 Additional Qualitative Analysis
Fine-tuned student simulators more closely match real students’ act and correctness distributions than prompting methods, but they still miss important behavioral patterns. Prompting methods often exploit majority-class correctness and overproduce information-seeking turns.
- Fine-tuned responses are often very short, which may contribute to their under-representation of the longer Seek Information turns.
- Fine-tuned methods more closely match real-student rates for Not Understanding, Acknowledge, and Off-Topic acts than prompting methods.
- Fine-tuned methods rarely produce Seek Information, while prompting methods overuse it and underproduce conversational acts such as Not Understanding, Acknowledge, and Off-Topic.
- Reasoning prompting improves several metrics over other prompting approaches but remains less accurate than fine-tuned models on act distributions.
- Prompting methods appear stronger on correctness partly because they more often predict the majority class, rather than matching the full ground-truth correctness distribution.
D.3 Results by Turn
Simulation difficulty varies across dialogue turns and metrics. Fine-tuned models benefit from simple opening turns, while limited context makes prompting and several later behavioral or knowledge judgments harder.
- The turn-level analysis truncates dialogues after 15 turn pairs, retaining 83% of dialogues to reduce high-variance results.
- All metrics vary across dialogue turns, with fine-tuned methods often finding the first turn easy because dialogues commonly begin with simple greetings.
- Prompting methods perform poorly on the first turn because they lack prior dialogue context.
- Acts are harder early in dialogues, whereas Knowledge Acquisition becomes more difficult through turns 0–6 before improving as context accumulates.
E Human Evaluation Details
The human evaluation used a reviewed, compensated annotation study to assess AI-generated middle-school mathematics dialogue responses. Its materials include recruitment, consent, interfaces, evaluation criteria, and qualitative examples.
- The human evaluation recruited four annotators, excluded one for consistently low-quality submissions, and paid each retained annotator $60 for two hours.
- Eligible participants were English-reading adults with prior mathematics teaching or tutoring experience in the United States or United Kingdom.
- Participants evaluated the appropriateness or realism of individual AI-generated responses in middle-school mathematics dialogues.
- The study materials included separate interfaces for evaluating ground-truth and simulated turns, alongside tables defining correctness and linguistic similarity.
- Qualitative examples compare ground-truth responses with outputs from all simulation methods and report corresponding metric scores.
G Prompts
The prompts are organized into annotation, evaluation-model, and student-model resources. They include prompts for labeling tutoring data, configuring evaluation models, and specifying multiple student simulation strategies.
- G.1 Annotations: Annotation prompts cover dialogue acts, correctness, knowledge components, question solutions, OCEAN personas, and Oracle summaries.
- G.1 Annotations: The dialogue-act annotation prompt supports labeling tutoring dialogue acts.
- G.1 Annotations: The correctness annotation prompt supports labeling correctness in tutoring interactions.
- G.2 Evaluation Models: Evaluation-model prompts specify fine-tuned classifiers for acts and correctness, alongside fine-tuned tutor, correctness-and-error prediction, and knowledge-tracing models.
- G.3 Student Models: The student simulation setup represents a student attempting a math problem while seeking help from a tutor.
- G.3 Student Models: Student-model prompts include fine-tuned, zero-shot, OCEAN, Oracle, in-context-learning, and reasoning student models.
- G.3 Student Models: The OCEAN student model prompt provides a persona-oriented student simulation configuration.
- G.3 Student Models: The Oracle, ICL, and Reasoning prompts provide distinct student-model configurations for oracle, in-context-learning, and reasoning approaches.