Source-linked AI summary
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan
TL;DR
AI tutoring evaluation needs reliable, scalable measures of pedagogical ability beyond answer correctness. MathTutorBench provides a holistic benchmark with automatic and learned metrics, finding a specialization-dependent trade-off between subject expertise and pedagogy. The benchmark is intended for rapid model comparison, while its scope is limited to high-school multi-step math and conversations of at most 10 turns.
Problem
Existing tutoring evaluations lack a reliable, easy-to-run measure that captures open-ended pedagogical quality and scales to comparisons with future models.
Method
MathTutorBench combines datasets and metrics across tutoring abilities and trains a reward model to score pedagogical quality in open-ended teacher responses.
Results
Subject expertise and pedagogy form a specialization-dependent trade-off: stronger solving ability often coincides with weaker pedagogy, while specialized tutors retain teaching ability longer.
Takeaways & Limitations
The benchmark supports quick, cost-effective model comparison as a proxy for selecting models for human studies, rather than replacing learning-outcome studies.
Takeaways & Limitations
The benchmark covers high-school multi-step math and excludes conversations longer than 10 turns, limiting coverage of other domains and long-term dependencies.
Abstract
from arXiv · showhide
Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.
1 Introduction
MathTutorBench addresses limitations in tutoring evaluation by combining holistic coverage of teaching skills with scalable automatic metrics. Its evaluation reveals that subject expertise and pedagogy can trade off, with specialization helping models sustain teaching quality in longer dialogs.
- LLM tutors need subject expertise, student-mistake diagnosis, and pedagogical techniques that promote student thinking rather than simply providing answers.
- Existing automatic metrics often measure word overlap or question-answering performance, while human evaluation is expensive and provides only static performance snapshots.
- MathTutorBench organizes evaluation into math expertise, student understanding, and teacher response generation, using standard metrics plus a reward model for pedagogical quality.
- Across open- and closed-weight models, subject expertise and pedagogical ability show a specialization-dependent trade-off: stronger solving often coincides with weaker pedagogy, while specialized tutors retain teaching ability longer.
- The benchmark and its released code and data aim to support quick, fair, and open comparison of tutoring models.
2 Related Work
Prior tutoring systems use prompts, fine-tuning, and preference alignment, but scalable evaluation remains difficult. MathTutorBench contributes a unified automatic approach for comparing models across key pedagogical aspects.
- LLM tutoring approaches include prompting pedagogical behavior, fine-tuning on pedagogical conversations, and aligning models with pedagogical preferences.
- Prompt-based methods can require tedious, sensitive prompt design, while collecting large-scale teacher-student data is expensive.
- MathTutorBench addresses the need for scalable evaluation by enabling quick comparison across key pedagogical aspects with simple, replicable metrics and real-teacher data.
- Automatic NLG metrics are noisy for open-ended tutoring, and specialized classifiers may cover only one teaching dimension or omit dialog history.
- Teacher annotations are time-consuming and hard to compare across trials, while learner studies are costly, time-consuming, and subject to ethics and privacy requirements.
3 Background
The paper models tutoring as generating the next teacher turn in a multi-turn interaction around a multi-step math problem. Its benchmark focuses on active-learning principles such as scaffolding, self-correction, and managing cognitive load.
- The benchmark studies teacher-student dialogues in which students solve multi-step problems with a single numerical answer and identifiable solution steps.
- Tutoring continues an existing dialogue by generating a teacher utterance conditioned on the history, optional problem knowledge, and teacher intent.
- The benchmark focuses on one-to-one, multi-turn interactions using scaffolding nudges, hints, and Socratic questioning to promote active learning.
- Its pedagogical principles require correctness, scaffolding instead of giving away answers, encouragement of self-correction, and avoidance of cognitive overload.
4 MathTutorBench
MathTutorBench evaluates math tutoring across expertise, student understanding, and pedagogical abilities using seven tasks and datasets. Its open-ended scaffolding evaluation uses preference-based reward modeling to score teacher responses.
- Benchmark scope: MathTutorBench measures Expertise, Student Understanding, and Pedagogical Abilities across seven math-tutoring tasks.Tasks use datasets, prompts, and metrics, with most problems sourced from GSM8k.
- Expertise and student understanding: Problem Solving measures final-answer accuracy, while Socratic Questioning tests guidance toward correct steps instead of directly giving answers.The benchmark also evaluates student-solution correctness and mistake location.
- Expertise and student understanding: Student Mistake Correction tests whether models recover from incorrect dialog steps while still producing a correct final answer.Unlike Problem Solving, this task conditions on dialog history containing an incorrect solution.
- Pedagogical abilities: Scaffolding Generation scores open-ended next-teacher utterances with a reward model, including a hard variant using longer conversation histories.Pedagogical Instruction Following tests prompts that request behaviors such as nudging students and asking guiding questions.
- Datasets: The benchmark combines Bridge’s 700 novice-teacher snippets with expert revisions and MathDial’s 2.9k teacher–simulated-student conversations.The combined MathDialBridge data are split for evaluation settings.
- Reward modeling: Criteria-based scoring aggregates binary predictions across n pedagogical criteria, but sparse features and large data requirements make this approach challenging.The cited example uses eight criteria, including guidance, actionability, and telling the answer.
- Reward modeling: A unified reward model replaces separate criterion classifiers by learning pairwise preferences between effective and less effective tutor responses.Preferred responses may use Socratic questions or probe understanding; rejected responses may reveal reference-solution content or exhibit fewer desired criteria.
5 Experiments
Experiments compare general, specialized tutoring, and math-reasoning models using teacher-response judgments and scaffolding evaluations. The benchmark includes independent Bridge-based validation, ablations, and multiple judge and reward-model baselines.
- Models: Experiments evaluate general LLMs, specialized tutoring models, and math-reasoning models, including Qwen2.5-Math-7B-Instruct.The evaluated set includes Llama, GPT-4o-mini, LearnLM, SocraticLM, ScienceTutor, and Qwen-based models.
- Teacher-response judgment: Pairwise ranking accuracy measures the proportion of expert teacher responses preferred over novice responses on an independent test set.The evaluation uses the expert-over-novice comparison defined in Equation 3.
- Scaffolding evaluation: The Scaffolding score uses 482 Bridge-derived examples in which novice responses were improved by expert teachers through error identification and pedagogical-strategy selection.The test set is designed to evaluate pedagogical response quality.
- Ablations: Table 3 reports Qwen2.5-1.5B-Instruct reward-model ablations, including training-instance totals and model additions, with the most accurate model selected for the Scaffolding score.Pretraining uses 20% of Ultrafeedback.
- Scaffolding models: The experiments group models by general, specialized tutoring, and math reasoning categories and compute win rate as reward-model preference for model responses over teacher responses.IF denotes Instruction Following.
6 Results
MathTutorBench reveals a trade-off between subject expertise and pedagogy, with tutoring specialization improving teaching but often reducing solving ability. Longer dialogs and pedagogical instruction following expose additional weaknesses, while pedagogically fine-tuned reward models outperform general judging approaches.
- Math expertise does not directly translate to student understanding or scaffolding, with strong problem-solving models often underperforming on scaffolding generation.
- Specialized tutoring models improve pedagogy but sacrifice some solving ability; SocraticLM improves scaffolding yet degrades across Student Understanding tasks.
- ScienceTutor trades lower math expertise for significantly better student-correction and pedagogical-instruction-following performance, while LearnLM achieves a more balanced profile.
- Longer dialogs make tutoring more difficult, as performance drops on hard splits and simpler questioning strategies become less effective; only LearnLM maintains consistent performance.
- Most models show limited pedagogical instruction following, although GPT4o and, to a lesser extent, ScienceTutor improve when instructions are explicit.
- Fine-tuning reward models on pedagogical preference data is essential: the resulting models outperform LLM judges and state-of-the-art RewardBench models.
- A reward model trained with human-scored pedagogical pairs and filtered MathDial conversations reaches 0.84 accuracy on a separate test set.
- Prompted and vanilla models separate expert from novice teacher responses poorly, supporting the need for high-quality pedagogical preference data.
7 Conclusion
MathTutorBench provides a quick, cost-effective way to assess the educational capabilities of LLM tutors. Its evaluations reveal trade-offs among expertise, understanding, and pedagogy, alongside diminishing performance in longer tutoring conversations.
- MathTutorBench enables quick, cost-effective assessment of tutoring models using lightweight automatic and learned metrics.
- The benchmark is intended to help select and compare models, not replace human studies that measure learning outcomes.
- Benchmark results show trade-offs among subject expertise, student understanding, and pedagogy, with diminishing results in longer tutoring conversations.
Limitations
The benchmark is scoped to high-school, multi-step math tutoring and does not cover all teacher functions or educational evaluation dimensions. Its conversational data also excludes dialogs longer than 10 turns.
- The benchmark’s insights are limited to high-school tutoring involving multi-step math problems.
- Conversations do not exceed 10 turns, limiting evaluation of very long educational dialogs with long-term dependencies.
- The study models 1:1 teacher–student tutoring using hints and nudges, not functions such as building rapport or trust.
- The benchmark omits some educational evaluation dimensions, including safety assessment of potentially harmful tutor responses.
Ethics Statement
The benchmark is released openly to support evaluation and improvement of tutoring models, while the authors note risks from reward hacking and exploitable data patterns. They encourage deployment with appropriate safeguards.
- The code and dataset are released under a CC-BY-4.0 license, following the licenses of the component datasets.
- The benchmark’s goal is to help improve tutoring models by balancing expertise, student understanding, and pedagogical guidance.
- Reward hacking could produce suboptimal tutoring behavior, while unknown data patterns could be exploited to obtain higher scores.
- The authors mitigate these risks by using varied data sources, mostly human-annotated, and recommend appropriate safeguards for deployment.
B.1 Details of Benchmarked Models
The benchmark evaluates specified versions of closed models and a set of open-weight models loaded from the Hugging Face Model Hub.
- The closed models are gpt-4o-mini-2024-07-18 and learnlm-1.5-pro-experimental.
- The open-weight set includes LLaMA3.2-3B-Instruct, LLaMA3.1-8B-Instruct, and Llama-3.1-70B-Instruct.
- It also includes CogBase-USTC/SocraticLM, princeton-nlp/Llemma-7B-32K-MathMix, and Qwen2.5-Math-7B-Instruct.
C.1 Implementation details
Implementation details cover model fine-tuning, benchmark execution, classifier training, and the prompts used for tutoring tasks and reward-based evaluation. The evaluation is designed for reproducible, relatively low-cost execution.
- C.1 Implementation details: All models are fine-tuned for one epoch with learning rate 1 · 10^-5, batch size 16, and AdamW on an NVIDIA A100 80GB GPU.Fine-tuning takes around 1 hour per model.
- C.1 Implementation details: The full Teacher response generation evaluation requires 2 954 generations and runs in under 10 minutes on one GH200 GPU.Throughput increases from 7.01 examples/sec at batch size 1 to 7.25 examples/sec at batch size 8.
- C.1 Implementation details: Criteria-based classifiers use ModernBERTbase models trained for three epochs on binarized MRBench data with AdamW, learning rate 1·10^-5, and batch size 16.
- C.1 Implementation details: Scaffolding reward scores classify responses from undesired behavior through neutral or useful behavior to best scaffolding practices, while novice responses are grouped into quartiles.
- C.1 Implementation details: The benchmark includes step-by-step solving and question-generation prompts for Problem-Solving and Socratic Questioning tasks.
- C.1 Implementation details: Reward-model and LLM-as-a-judge prompts assess pedagogical quality using problems, reference solutions, dialog history, and teacher utterances.