PaperlayerView exact revision on arXiv

Source-linked AI summary

MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems

Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

arXiv:2305.14536v2cs.CL

TL;DR

Dialogue tutoring research lacks sufficiently large, high-quality datasets, while existing models can give incorrect feedback or reveal solutions too early. The paper pairs human teachers with LLM-simulated students to construct MathDial, a 3k-dialogue math tutoring dataset with scaffolding and grounding annotations. Finetuned smaller models achieve similar student-solving success to prompted LLMs while directly telling answers less often, though student simulation and generalization remain limitations.

  • Problem

    Dialogue tutoring research lacks sufficiently large, high-quality datasets, while current models can provide incorrect feedback or reveal solutions too early.

  • Method

    The paper pairs human teachers with LLM-simulated students and collects math tutoring dialogues using scaffolding moves and grounding annotations.

  • Results

    Finetuned smaller models achieve similar student-solving success to prompted LLMs while keeping the direct telling rate lower.

  • Takeaways & Limitations

    MathDial provides a publicly released dataset for training and evaluating tutoring models on equitable, faithful guidance rather than solution generation alone.

  • Takeaways & Limitations

    LLM-simulated students may under- or over-represent certain kinds of human student confusions.

Abstract

from arXiv · show

While automatic dialogue tutors hold great potential in making education personalized and more accessible, research on such systems has been hampered by a lack of sufficiently large and high-quality datasets. Collecting such datasets remains challenging, as recording tutoring sessions raises privacy concerns and crowdsourcing leads to insufficient data quality. To address this, we propose a framework to generate such dialogues by pairing human teachers with a Large Language Model (LLM) prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues grounded in multi-step math reasoning problems. While models like GPT-3 are good problem solvers, they fail at tutoring because they generate factually incorrect feedback or are prone to revealing solutions to students too early. To overcome this, we let teachers provide learning opportunities to students by guiding them using various scaffolding questions according to a taxonomy of teacher moves. We demonstrate MathDial and its extensive annotations can be used to finetune models to be more effective tutors (and not just solvers). We confirm this by automatic and human evaluation, notably in an interactive setting that measures the trade-off between student solving success and telling solutions. The dataset is released publicly.

1 Introduction

MathDial addresses gaps in dialogue tutoring by pairing human teachers with an LLM-simulated student and grounding tutoring in scaffolding and annotations. The resulting dataset supports models that tutor more equitably and faithfully, with finetuned smaller models matching ChatGPT’s solving success while revealing solutions less often.

  • Scaffolding questions and grounding annotations provide students opportunities to learn, think, and explore rather than receiving leaked solutions.
  • High-quality tutoring datasets remain scarce because existing data can be small, pedagogically weak, noisy, or difficult to collect without privacy and quality concerns.
  • Current tutoring models often provide incorrect feedback or reveal solutions too early despite high accuracy on math word problems.
  • MathDial pairs human teachers with an LLM simulating student profiles, errors, and confusions judged representative of real students.
  • MathDial is grounded in math word problems and student confusions, creating a challenging testbed for faithful and equitable tutoring models.
  • Finetuned smaller open-source LLMs achieve similar student-solving success as ChatGPT while telling solutions less often.

2 Background & Related Work

Prior dialogue-tutoring datasets and collection methods face limitations in quality, grounding, scale, and noise. MathDial adapts the Wizard-of-Oz paradigm by combining expert human teachers with LLM-based student simulation and scaffolding-oriented tutoring research.

  • Crowdsourced and scraped dialogue collection can introduce hallucinations, annotation errors, low lexical diversity, cost, and insufficient annotator quality.
  • The collection pipeline oversamples and frequency-sorts student confusions before synchronously pairing a teacher with an LLM-simulated student.
  • MathDial adapts the Wizard-of-Oz paradigm by using human teachers as experts collaborating with an LLM.
  • Its tutoring design draws on research showing that questioning, deep reasoning, elaboration, and challenging prior contributions can benefit student learning.
  • Existing tutoring corpora commonly suffer from insufficient tutoring quality, missing grounding information, small sizes, or noisy classroom settings.

MATHDIAL Collection Pipeline

MATHDIAL pairs expert teachers with an LLM simulating students and their misconceptions to collect grounded tutoring dialogues. The resulting annotations support analysis of student plausibility, teacher strategies, dialogue quality, and confusion resolution.

  • Collection framework: Human teachers roleplay tutors while an LLM simulates students with one selected incorrect solution and one of six misconception profiles.Dialogues are grounded in GSM8k multi-step math word problems, with teachers accessing the problem, correct step-by-step solution, and initial confusion.
  • Collection framework: Teachers must select a current move before responding, producing sequences of scaffolding strategies that end when the problem is solved or time expires.The dataset records the problem, solution, confusion step, resolution status, and equation-form student and step-by-step solutions.
  • Dataset quality: MATHDIAL is larger than related datasets, has higher bigram-entropy diversity than CIMA and TalkMoves, and has uptake similar to CIMA while surpassing other compared datasets.Its diversity is similar to NCTE and TSCC, which are classroom and one-to-one tutoring transcripts, respectively.
  • Student simulation: Teachers rated the majority of simulated student generations as plausible along typical sixth-grader confusion and interaction dimensions.These judgments are released as annotations, alongside distinctions between conceptual and arithmetic errors.
  • Teacher strategies: 37% of teacher utterances are Focus moves, while Generic and Probing follow; Telling is the rarest move.Focus directs students toward the solution without giving it, whereas Probing develops relevant reasoning and world knowledge.
  • Teacher strategies: Focus and Probing dominate early scaffolding, while Telling becomes more likely as conversations progress.Teachers typically begin generically, then focus or probe specific problem elements before increasingly revealing information.
  • Outcomes: Teachers reported resolving confusion in almost 89% of conversations, mainly through scaffolding in about 75% and by revealing most of the answer in around 14%.Unresolved conversations remain useful for training classifiers to identify cases requiring human intervention.

MATHDIAL

MATHDIAL is used to train and evaluate tutor-response models with dialogue history, grounding information, and teacher-move conditioning. The experiments compare finetuning and prompting, test seen versus unseen problems, and evaluate interactive tutoring trade-offs.

  • Modeling task: Tutor-response generation models produce follow-up teacher turns conditioned on dialogue history, annotated information, and a selected teacher move.Grounding information includes the math word problem, step-by-step solution, and student solution attempt.
  • Main findings: Finetuned models on MATHDIAL can outperform much larger prompted models, although generalization remains weaker and scaling alone does not guarantee improvement.The table caption summarizes the main response-generation findings.
  • Evaluation setup: The dataset uses an 80% training split and 20% test split, with about 60% of test problems seen during training and 40% unseen.The unseen split tests generalization to new problems.
  • Evaluation setup: Models are evaluated with sBLEU, BERTScore, and token-level F1, including faithfulness comparisons between generated utterances and the math word problem.The evaluation also compares generated responses with annotated teacher responses.
  • Grounding ablation: The step-by-step solution is the most valuable grounding input in the reported ablation.The question and incorrect solution are less crucial but recur frequently in the dialogue.

6 Results

Results show that finetuning on MATHDIAL improves tutoring quality relative to prompting larger models and supports a better balance between student success and revealing solutions. However, performance remains limited on unseen and longer problems.

  • Tutor response generation: Larger model size alone does not consistently improve tutoring metrics, while instruction tuning improves Flan-T5 over T5.Smaller models may rely more on grounding information, whereas instruction tuning improves performance.
  • Tutor response generation: Finetuned models show a large performance gap between seen and unseen problems, indicating weak generalization to new problems.Finetuned models generally outperform zero-shot prompted GPT on automatic metrics, but the validity of those metrics is questioned.
  • Grounding information: Step-by-step solutions are crucial grounding information for tutor response generation, while questions and incorrect solutions are less important.Questions and incorrect solutions are often repeated during the dialogue, but the step-by-step solution has the strongest effect.
  • Tutor response generation: Finetuning smaller models on MATHDIAL improves correctness and equitable tutoring compared with prompting larger language models.Human evaluation reports gains in correctness and equitable tutoring after finetuning.
  • Interactive evaluation: Flan-T5 achieves a similar success@5 rate to baselines while telling solutions less often in interactive tutoring.NEXTSTEP has the lowest success rate but never tells solutions, whereas ChatGPT has high success and the highest telling rate.
  • Interactive evaluation: The gap between Flan-T5 and ChatGPT success rates stems mostly from longer problems, and no model matches ground-truth success.The limitation is most evident in the initial steps of longer problems.

7 Conclusion

The paper introduces MATHDIAL, a large, pedagogically rich dataset and collection framework for tutoring math word problems. Finetuned models improve over prompted larger LLMs, but still need stronger reasoning and generalization.

  • Dataset and framework: The paper introduces a semi-synthetic framework that pairs human teachers with LLM-simulated students to collect pedagogically rich tutoring dialogues.The dialogues follow equitable tutoring practices and learning-science research on scaffolding student understanding.
  • Dataset and framework: MATHDIAL contains approximately 3k tutoring conversations grounded in GSM8K math word problems.The dataset consists of one-on-one tutoring conversations grounded in multi-step math problems.
  • Results: Finetuned smaller models can significantly surpass much larger prompted LLMs on tutor response generation.In interactive tutoring, finetuned models achieve similar student-solving success while directly telling solutions less often.
  • Open challenges: Models still require better reasoning over student solutions and better generalization to unseen problems.These remain open challenges despite the reported gains from finetuning on MATHDIAL.
  • Implications: The dataset provides a larger testbed for studying effective dialogue tutors at scale.It is intended to support research on one-on-one tutoring and better tutoring models.

8 Limitations

The dataset’s limitations concern simulated student behavior, interactional fidelity, domain scope, teaching coverage, and the short-term nature of its learning measure.

  • Student simulation: LLM-simulated student confusion may under- or over-represent certain kinds of real student confusions.The authors identify the simulator’s limited understanding of human learning as a key dataset limitation.
  • Interactional fidelity: Teachers interacting with an LLM student may behave differently than they would in classrooms, and motivation may decline when the student is artificial.The authors also connect this setup to possible lower data quality and call for better LLM-based student models.
  • Domain scope: The collection methodology was instantiated only for math reasoning, so additional domain-specific datasets are needed to assess broader effectiveness.The paper explicitly frames domain expansion as necessary for further generalization.
  • Teaching coverage: The teaching taxonomy covers only a subset of common teaching moves and omits goals such as metacognitive support, rapport, and drawing-based instruction.Text tutoring also limits teachers’ use of additional instructional practices.
  • Evaluation scope: Immediate problem-solving success does not capture all student learning, making long-term learning a desired focus for evaluation.The authors note that scaffolding questions requiring conceptual understanding may support deeper and longer-term learning even when immediate answers are incorrect.

B Problem and confusion selection

The pipeline selects challenging GSM8k problems with plausible wrong solutions, filters simple or unsuitable errors, and constructs varied student confusion profiles for annotation.

  • Problem selection: The pipeline samples 50 reasoning paths per problem and uses majority voting over final answers to identify disagreements with the ground truth.Problems are retained with at most four solution steps.
  • Filtering: Rounding errors within 0.1 of the original solution and simple calculator-detectable arithmetic inconsistencies were discarded.The filters target confusions considered less informative for learning.
  • Problem selection: 5684 GSM8k problems were queried, yielding 2313 with at least one wrong solution after excluding problems exceeding five solution steps.This establishes the initial pool before solution-length and answer-distribution filtering.
  • Filtering: 1379 wrong solutions remained after filtering solutions to 300–500 characters, and the final set contained 1131 problems after removing near-unanimous wrong-answer cases.The final filter removed problems where 49 or 50 of 50 proposed solutions shared the same wrong answer.
  • Student confusion: Student personas represent varied algebra misconceptions, including difficulties with procedures, principles, problem interpretation, relevant information, operations, and strategy selection.The dataset uses student names and pronouns alongside these characteristics to represent students from varied backgrounds.
  • Student confusion: Manual screening found simple arithmetic errors and inconsistent student behavior as common student-model problems, which are reflected in teacher quality ratings.The authors state that further analysis is needed to understand this behavior more finely.

D Data collection interface

Teachers analyze simulated student errors, then conduct time-bounded scaffolded tutoring conversations through an interface that records teacher moves and post-conversation judgments.

  • Collection workflow: Each annotator receives a student and math word problem, analyzes the submitted solution, conducts tutoring, and completes a questionnaire about the conversation and error.Teachers are instructed to scaffold understanding rather than simply correct the solution.
  • Error annotation: Teachers categorize errors as extra or missing quantities, calculator-solvable calculation errors, missing or wrong factual knowledge, question misunderstanding, or other cases.They also record cases where a student reached the correct solution but continued further.
  • Interface: The interface presents the problem, correct and student solutions, conversation history, response-category controls, and a text field that invokes the student model after sending.The teacher first compares solutions and marks the exact error step.
  • Conversation procedure: Conversations end when the student solves the problem correctly or after 10 minutes.This creates a fixed practical boundary for the tutoring interaction.
  • Quality control: Annotators study tutoring best practices, pass a task-understanding test, and receive manual quality checks during training.The checks assess conversation quality and use of diverse scaffolding questions.
  • Teacher moves taxonomy: Focus directs students toward a solution without giving it, Probing develops relevant reasoning or world knowledge, Telling supplies solution parts, and Generic covers other conversational elements.Focus and Probing are identified as the most productive scaffolding questions for long-term learning, while Telling is mainly for stuck students.
  • Collection efficiency: The human-teacher/LLM-student setup creates conversations in about 6 minutes per 7+ turn dialogue and was found more efficient and performant than human-human or fully synthetic alternatives.The authors describe this approach as scalable because one user can create an entire conversation with the LLM.

E Interactive evaluation of tutoring

The interactive evaluation uses a fixed simulated student and teacher prompts that favor empathetic questioning over direct answers.

  • Evaluation setup: All evaluated setups use an InstructGPT student model named Kayla, with the teacher’s opening utterance fixed across conditions.The fixed opening asks Kayla to explain her solution.
  • Evaluation setup: The ChatGPT teacher prompt casts the tutor as empathetic and discourages direct answers in favor of questions that help students understand concepts.The prompt frames tutoring as collaborative problem solving using the given math problem and correct solution.

F Human Evaluation Protocol

Human evaluation rates tutoring responses for coherence, correctness, and equitable tutoring using defined scales, while conversations include student solutions and error categories. A pilot study also examined synthetic data generation for structural and factual inconsistencies.

  • Evaluation dimensions: Annotators rated coherence, correctness, and equitable tutoring as dimensions of tutoring quality.Coherence concerns contextual follow-up, correctness covers factual and mathematical accuracy, and equitable tutoring provides space for student learning.
  • Rating scales: Coherence and equitable tutoring used a 3-point Likert scale, whereas correctness used a binary scale.The Likert scale ranged from 1 (poor) to 3 (very good).
  • Evaluation setup: The ChatGPT evaluation prompt matched the interactive tutoring scenario and additionally included the student solution.This setup supplied the model with the student’s attempted solution during evaluation.
  • Conversation materials: MATHDIAL conversation examples mark teacher and student turns and ground each dialogue in correct and student solutions.Bold text identifies error categories for the reader.
  • Pilot observation: The pilot found that InstructGPT’s synthetic data generation repeatedly used next-step questions and showed factual and step-order inconsistencies.The figure marks the recurring question structure and inconsistencies separately.
Loading 2305.14536v2…