PaperlayerView exact revision on arXiv

Source-linked AI summary

MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin-Xia Huang

arXiv:2610.31679v1cs.CL

TL;DR

Evaluating LLM math tutors requires assessing the full detect–diagnose–remediate pipeline and clarifying what tutoring scores measure. MISCONCEPTDIALOGUE evaluates that pipeline with simulated students and separates tutor-information conditions from scoring criteria. On identical dialogues, scoring criterion shifted measured remediation success substantially, while supplied diagnostic information showed little evidence of reliably changing tutor behavior under the tested format.

  • Problem

    Existing studies assess tutoring stages in isolation, while evaluation criteria may measure distinct targets such as answer strings, numerical equivalence, and conceptual understanding.

    Source: §1 · pp. 1–2

  • Method

    MISCONCEPTDIALOGUE evaluates five open-weight LLM tutors across more than 5,000 simulated-student dialogues, varying pipeline information and scoring lenses.

    Source: §1 · p. 2

  • Results

    39.2 pp separated benchmark-style exact-match scoring from human-calibrated LLM-as-judge Remediation Success Rate on identical dialogues, exceeding the combined effects of tutor model, strategy, and supplied diagnostic information.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Under the tested diagnosis-presentation format, tutors showed little evidence of reliably conditioning on supplied diagnostic labels, while measured success depended strongly on the evaluation criterion.

    Source: §1 · p. 2, §6 · p. 9

    GPT-5.4 generated 47% of remediation scenarios, and use of an OpenAI-family default judge could inflate RSR; the reported checks bound but do not eliminate this risk.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student's error, diagnose the underlying misconception, and remediate it through multi-turn dialogue - and can we trust the numbers that report how well it does? We address both with MISCONCEPTDIALOGUE, a fully automated framework that runs the full pipeline on five open-weight LLM tutors (7B-14B) interacting with LLM-simulated students over 5,000+ dialogues - a deliberately simulated protocol (paired conditions, a wrong-diagnosis control unsafe with real learners, scale) whose results do not establish real-student learning. On measurement, our headline finding is uncomfortable for the field: benchmark-style exact-match scoring common in standard math NLP - a lower-bound stress test - reports 42.1% measured Remediation Success Rate on identical dialogues where a human-calibrated LLM-as-judge reports 81.3%, a 39.2 percentage-point spread larger than the combined effects of tutor model, teaching strategy, and supplied diagnostic information. This is not a single-judge artifact: four independent LLM judges and 200 human annotations converge to within a few points (kappa up to 0.91). On capability, the verdict is equally striking: the score barely changes whether we tell the tutor the student's actual misconception or a deliberately swapped one - under the tested diagnosis-presentation format, these tutors show little evidence of reliably conditioning on the diagnostic information they are handed. For LLM tutoring evaluation to mature, multi-criterion scoring with multi-judge convergence and human anchoring should become standard practice - and the measurement question deserves as much scrutiny as the capability question it claims to answer.

1 Introduction

MISCONCEPTDIALOGUE addresses gaps in evaluating the full detect–diagnose–remediate pipeline and in determining what tutoring evaluation scores measure. Its results show that scoring criterion can shift measured remediation success more than the tested modeling choices.

1 Introduction

  • Student misconceptions are coherent patterns of incorrect reasoning that resist correction by merely presenting correct answers.The cited learning-science account holds that remediation requires cognitive conflict and that different misconception categories can require different interventions.Source: §1 · p. 1
  • Prior evidence on LLM tutors includes reported gains and pedagogical alignment, but benchmark studies also find gaps versus human tutors and limited adaptation to learner cognitive states.The paper frames capability and measurement as two unresolved questions for tutors deployed at scale.Source: §1 · p. 1
  • Existing benchmarks assess isolated stages, leaving the full pipeline under-evaluated in multi-turn dialogue.Detection, diagnosis, and remediation are interdependent: errors can lead to missed interventions, inappropriate strategies, or answer-revealing instead of understanding-guiding tutoring.Source: §1 · pp. 1–2
  • Tutoring scores may measure different targets because final-answer strings, numerical equivalence, and conceptual understanding are related but distinct.MISCONCEPTDIALOGUE uses simulated students for scale and experimental control, so its capability claims are scoped to that protocol.Source: §1 · p. 2
  • 42.1% Remediation Success Rate under benchmark-style exact-match scoring versus 81.3% under a human-calibrated LLM-as-judge on the same dialogues, a 39.2 pp gap.The paper treats exact match as a lower-bound stress test, not a normative tutoring metric.Source: §1 · p. 2
  • Four LLM judges converged within 6 pp (Fleiss κ=0.85), while human-annotated subsets showed judge–human agreement of κ=0.79–0.91 on clean cases.Judges remained informative on borderline cases (κ=0.74–0.87).Source: §1 · p. 2

4. Diagnosis–detection divergence: One stage

Detection strength does not reliably indicate whether a tutor can identify the underlying misconception; mistake identification and misconception identification are distinct capabilities.

4. Diagnosis–detection divergence: One stage

  • Across all five models, detection strength failed to predict diagnosis accuracy, exposing a category error in benchmarks that conflate mistake identification with misconception identification.Under the tested protocol, a wrong-diagnosis control also provided little evidence that tutors reliably condition on supplied diagnostic labels.Source: §4 · p. 2

2 Related Work

Prior work studies misconception diagnosis, dialogue tutoring, and evaluation validity, but leaves important questions about their integration unresolved. MISCONCEPTDIALOGUE positions its evaluation around those gaps.

2.1 From Detection to Diagnosis

  • Prior work separates structured misconception diagnosis from dialogue evaluation, and whether contextual detection predicts fine-grained diagnosis within one tutoring pipeline remains open.Detecting an incorrect response is not the same as identifying the misconception that remediation must address.Source: §2.1 · pp. 2–3

2.2 LLM Tutoring and the End-to-End Evaluation Gap

  • Prior LLM tutoring evaluations are component-centered, leaving under-tested whether an integrated detect–diagnose–remediate pipeline improves over direct tutoring.Existing work reports remaining gaps in adaptivity versus traditional intelligent tutoring systems and performance versus human tutors.Source: §2.2 · p. 3

2.3 The Measurement Problem

  • LLM judges have documented biases, while simulated students raise authenticity, emotional-range, and cognitive-diversity concerns; their impact on tutoring measurement remains underexplored.The paper identifies uncertainty about how closely simulated learners track real learners as an active measurement question.Source: §2.3 · p. 3
  • The framework separates tutoring, tutor-information interventions, and scoring lenses so their effects can be examined independently.Figure 1 depicts the D-G-R pipeline, information conditions, and three evaluation lenses; the same 2,250 dialogues receive RSR scores of 42.1% to 81.3% across lenses.Source: Figure 1 · p. 3
  • Together, these gaps leave unclear how much measured Remediation Success Rate reflects tutoring behavior rather than evaluation artifacts.The paper introduces MISCONCEPTDIALOGUE to address the gaps jointly.Source: §2.3 · p. 3

3 The MisconceptDialogue Framework

MISCONCEPTDIALOGUE separates the tutoring pipeline, information supplied to tutors, and evaluation criteria. It applies a defined misconception taxonomy and remediation strategies, then scores dialogue outcomes with multiple criteria.

3 The MisconceptDialogue Framework

  • The framework separates the D-G-R tutoring pipeline, pipeline-information interventions, and evaluation lenses applied to the resulting dialogues.This separation allows experiments to vary scoring criteria or tutor information independently.Source: §3 · p. 3
  • The 55 educator-validated algebra misconceptions form a closed-set taxonomy, instantiated as 106 remediation scenarios with held-out verification problems.Scenarios vary problem context across factual, procedural, and conceptual misconceptions; 53% use benchmark items and 47% GPT-5.4-generated variants.Source: §3 · p. 4
  • Five remediation strategies are defined: Counterexample, Analogy, Cognitive Conflict, Direct Instruction, and Socratic Questioning.Experiment 3 fixes one strategy; Experiment 4 lets the tutor choose a strategy from the diagnosis.Source: §3 · p. 4
  • Remediation Success Rate measures scenarios where a simulated student solves a held-out verification problem, scored by exact match, numerical extraction, or a human-calibrated LLM judge.Four judge models span three vendor families; the framework also reports Answer Reveal Rate, detection Macro F1, and diagnosis Accuracy@1.Source: §3 · p. 4

4 Experiments

Across these experiments, measured remediation success varies sharply with scoring criterion, while the end-to-end pipeline shows no significant RSR effect under the simulated protocol. Detection and diagnosis diverge, and LLM judges align closely with human consensus on annotated subsets.

4.1 Models and Datasets

  • The study evaluates five open-weight 7B–14B tutors across detection datasets, a 55-misconception diagnosis benchmark, and a 106-scenario remediation bank.Remediation tests five strategies across three misconception types using simulated students.Source: §4.1 · p. 4, §4.4 · p. 5

4.2 Experiment 1: Context-Aware Detection

  • Multi-turn context improved F1 on all-incorrect datasets but reduced it by 0.026–0.163 on mixed-label MathDial, alongside a shift toward incorrect predictions.As context grew, mean incorrect predictions rose from 55.8% to 79.7%, while correct-class recall fell from 0.444 to 0.202.Source: §4.2 · pp. 4–5, Table 1 · p. 6

4.3 Experiment 2: Diagnosis Accuracy

  • Detection and diagnosis diverged: Mistral-7B had the strongest mixed-label detection F1 (0.492) but best-case diagnosis accuracy at most 0.255, while Qwen3.5-9B led diagnosis at 0.545.GRR raised Mistral diagnosis accuracy from 0.077 to 0.255 but remained below Qwen3.5-9B zero-shot at 0.545.Source: §4.3 · p. 5, Table 2 · p. 6

4.4 Experiment 3: Remediation and Evaluation Methodology

  • 42.1% strict-match RSR versus 77.8–84.4% LLM-judge RSR on the same 2,250 dialogues shows that measured success depends strongly on the scoring criterion.Strict match and numeric extraction are nearly equivalent (42.1% vs. 45.8%).Source: §4.4 · p. 5, Table 3 · p. 6
  • Of 949 strict-fail/judge-pass dialogues, 23.4% also passed numeric extraction and 76.6% did not, separating string-matching brittleness from broader construct-credit differences.The authors attribute roughly three-quarters of the scoring gap to construct-credit differences and the remaining quarter to surface-matching brittleness.Source: §4.4 · p. 6
  • Conceptual cases had the highest judge RSR (89.9%) and factual cases the lowest (73.2%), while strategies clustered within 6.7 pp; simulated-student recovery varied by misconception type.Persona-only RSR was 6.6%, and conceptual gains exceeded procedural gains by 25.6 pp.Source: §4.4 · p. 6

4.5 Experiment 4: End-to-End Pipeline

  • Pipeline conditions showed no significant RSR effect (p=0.263); a same-type wrong diagnosis scored 80.8%, statistically indistinguishable from gold diagnosis or direct tutoring.The authors note that the swap tests fine-grained label uptake while holding coarse misconception type fixed.Source: Table 4 · p. 7, §4.5 · p. 7

4.6 Human Evaluation

  • Four LLM judges tracked human consensus within 1–3 pp on clean cases and 2–7 pp on borderline cases, with item-level κ of 0.79–0.91 and 0.74–0.87, respectively.Three blinded annotators rated 100 clean and 100 borderline dialogues.Source: Table 5 · p. 8, §4.6 · p. 8

5 Discussion

On identical dialogues, the measured remediation rate depends more on the scoring criterion than on tutor or pipeline choices. The discussion pairs this measurement finding with a qualified null result on diagnostic-information uptake and recommends reporting criterion disagreement.

5 Discussion

  • The scoring criterion shifts RSR from 42.1% to 81.3% on identical dialogues, a 39.2 pp difference larger than the single-digit effects of model, strategy, pipeline condition, and judge identity.Two non-OpenAI judges reproduce a 33.3–39.2 pp gap, and the three main judges converge at Fleiss κ=0.85.Source: §5 · p. 8
  • Strict match is a stress test, not a normative tutoring metric; human agreement on clean dialogues supports interpreting LLM-judge scores as understanding judgments.Item-level agreement between LLM judges and human consensus reaches κ up to 0.91.Source: §5 · p. 8
  • Perfect pipeline information does not detectably improve remediation under this protocol (p=0.263), and the null tests fine-grained label uptake rather than diagnosis’s value wholesale.The authors note that tutor non-use, simulator recovery, or judge insensitivity could explain the result; the single null cannot distinguish these readings.Source: §5 · p. 8
  • The paired audit fixes dialogues while varying scoring in Exp. 3, then fixes the evaluator while varying tutor information in Exp. 4.This design isolates criterion effects from tutor outputs and pipeline-information effects from evaluator choice.Source: §5 · p. 8
  • LLM-judge RSR settles near 79% across independent pipeline conditions, motivating reports with at least two automated criteria, their disagreement, and a human baseline.Exp. 3 reports 78–84% and Exp. 4 reports 74–78%; disagreement sets should be audited rather than collapsed into one score.Source: §5 · pp. 8–9

6 Conclusion

MisconceptDialogue examines tutoring capability and measurement across five open-weight LLM tutors. Its conclusion is that scoring criteria strongly shift measured success, while diagnostic information yields no detectable advantage under the tested conditions; real-student study remains the next step.

6 Conclusion

  • On the same 2,250 dialogues, changing the scoring criterion moves RSR from 42.1% to 81.3%, a 39.2 pp gap exceeding the combined effects of tutor model, teaching strategy, and supplied diagnostic information.Four judges spanning three vendor families reproduce the gap, and 200 human annotations anchor it with judge–human κ=0.74–0.91.Source: §6 · p. 9
  • Structured pipeline information yielded no detectable advantage over direct tutoring (GLMM p=0.263), and the wrong-diagnosis control found little evidence of conditioning on diagnostic labels.The conclusion scopes this capability finding to the tested protocol.Source: §6 · p. 9
  • The authors argue that tutoring evaluation should treat scoring criteria as a first-class object of study, with real-student research as an essential next step.The calibrated instrument developed in this protocol is described as what a real-student study will need first.Source: §6 · p. 9

Limitations

The limitations chiefly bound generalization beyond this protocol: its domain and misconception taxonomy are narrow, its evaluation relies on simulated students and limited calibration, and some null findings remain format- or scale-specific.

Limitations

  • The 55 educator-validated middle-school algebra misconceptions are a small slice of real curricula, and more ambiguous domains would require adapted criteria.The paper identifies external validity to real classrooms, broader domains, and larger model families as the primary limitation.Source: Limitations · p. 9
  • BEA and Bridge contain only incorrect responses, while the Exp. 4 C4 detection rate of 96.8% is recall, not specificity.Additionally, GPT-5.4 generated 47% of remediation scenarios and the OpenAI-family default judge could inflate RSR through in-family coherence; cross-vendor replication and clean-set human RSR bound but do not eliminate that risk.Source: Limitations · p. 9
  • Five 7B–14B open-weight tutors and two stronger proprietary tutors were evaluated; the suggestion that 70B+ systems behave similarly remains unverified.Model identity contributes negligible variance (ICC<1%).Source: Limitations · p. 9
  • Exp. 3 has 30 dialogues per cell and moderate per-cell power; Exp. 4’s null leaves benefits near 5 pp possible, and human calibration covers two quality-stratified subsets of 100.TOST accepts equivalence at δ=10 pp for all 15 pairs but only three at δ=5 pp; power is 80% for effects ≥7 pp.Source: Limitations · p. 9
  • GPT-5-mini serves as both simulated student and one of four judges, while human calibration only partially anchors structural biases shared across LLM judges.Cross-family validation, non-OpenAI replication, and human calibration on 200 dialogues probe this dual-role risk.Source: Limitations · p. 9
  • The diagnosis is injected in one prompt format, so the wrong-diagnosis control tests label conditioning only under that format; strategy invariance could reflect a uniformly persuadable simulator.Separating the strategy interpretations requires real students.Source: Limitations · p. 9

Ethical Considerations

The evaluation uses simulated students and external datasets, not real student participants. Its results may overestimate classroom effectiveness, and the authors recommend human expert review before autonomous deployment in high-stakes settings.

Ethical Considerations

  • The study involved no real students, and simulated learners may overestimate classroom effectiveness because they cannot fully capture diverse reasoning, language backgrounds, or affective responses.The authors recommend human expert review before autonomous deployment, particularly in high-stakes educational contexts.Source: Ethical Considerations · p. 10

A Prompt Specifications

  • Prompts separately specify mistake detection, misconception ranking, strategy-constrained remediation, conceptual-understanding judging, simulated-student behavior, and dialogue-quality screening.The judge distinguishes conceptual grasp from numerical correctness, while quality screening rates coherence, role adherence, educational value, and verification relevance.Source: A Prompt Specifications · p. 13

B Simulated Student Profiles

  • Six simulated-student profiles cross three knowledge levels—low, medium, and high—with cooperative or resistant behavior.Each profile also specifies the student’s misconception.Source: B Simulated Student Profiles · p. 13

C Dataset Statistics

D Diagnosis Accuracy by Misconception Type

  • Procedural misconceptions are easier to diagnose than conceptual ones for most models, but the type-wise results are descriptive because counts are imbalanced.Mistral is weakest overall and improves under retrieval-style conditions, while Qwen3.5 is strongest overall.Source: D Diagnosis Accuracy by Misconception… · p. 13, Table 7 · p. 14

E Dataset Label Distributions

  • MathDial has 1,696 correct and 566 incorrect responses (75%/25%); on all-incorrect datasets, context can inflate accuracy and F1 by favoring incorrect classifications.BEA 2025 and Bridge contain only incorrect responses.Source: E Dataset Label Distributions · pp. 13–14

F Experimental Hyperparameters

G Diagnosis Experiment Methodology

The section details diagnosis experiments and human evaluation, alongside statistical checks of scoring and evaluator comparisons. With evaluation context aligned, two judges report closely tracking remediation success rates.

G Diagnosis Experiment Methodology

  • Diagnosis experiments rank 55 labels, compare raw-query retrieval with GRR, and uniformly regex-correct IDs; few-shot results are treated as format-conditioning baselines because labels may overlap.GRR generates a diagnosis, retrieves taxonomy candidates, then reranks them.Source: G Diagnosis Experiment Methodology · pp. 14–15

H Human Evaluation Details

  • Human calibration uses clean and borderline dialogue subsets with a four-dimension rubric; inter-annotator agreement is moderate, and naturalness ratings are descriptive rather than validated.The subsets contain 100 dialogues each, and only a “Yes” on correct understanding maps to “correct” for automated-score comparisons.Source: H Human Evaluation Details · p. 15

I Human–Judge Disagreement Examples

  • The judge and human consensus disagreed on three of 100 clean dialogues: two false positives and one false negative.Examples include crediting method comparison or formula use despite an unresolved misconception, and rejecting a conceptually correct limit explanation for an incomplete computation.Source: I Human–Judge Disagreement Examples · p. 16

J Statistical Test Rationale: McNemar

  • On the paired 2,250-dialogue comparison, 949 strict-match failures were judge passes versus 67 in the opposite direction, yielding χ2=765.7.The paired bootstrap 95% confidence interval for the gap is [36.6, 41.8] pp.Source: J Statistical Test Rationale: McNemar · p. 16

K TOST Equivalence Testing

  • At a 5 pp equivalence margin, only three priority pairs are equivalent; at 10 pp, all 15 pairwise comparisons are equivalent.The comparisons use TOST on two independent proportions, with n=530 per condition.Source: Table 10 · p. 16, K TOST Equivalence Testing · p. 16

L Cross-Family Evaluator Validation

M Strategy Fidelity Analysis

Strategy-constrained prompts produce distinguishable linguistic patterns, but qualitative examples and persona comparisons show that remediation outcomes also vary with scenario amenability and simulated-student behavior.

M Strategy Fidelity Analysis

  • Socratic dialogue has the highest self-marker ratio (0.616); strategy-constrained prompts produce statistically distinct linguistic patterns, though some strategies overlap in general tutoring language.The marker-ratio comparison is significant at p < 10^-6.Source: M Strategy Fidelity Analysis · p. 17, Table 12 · p. 17

N Strategy Invariance: Qualitative Examples

  • The independence misconception succeeds across all 25 model-strategy cells, while procedural double-counting fails across strategies, suggesting scenario amenability dominates these examples.In the failure example, students solve the tutored problem but fail verification on a novel scenario, indicating no transfer of the underlying skill.Source: N Strategy Invariance: Qualitative… · pp. 17–18

O Persona Decomposition

  • 74.7 pp B→C lift (6.6% to 81.3%) shows tutoring moves simulator scores, while the 91.5 pp A→B drop argues against a yes-machine or simple persuadability ceiling.The tutoring lift is type-dependent: 63.6 pp for conceptual and 38.0 pp for procedural scenarios.Source: O Persona Decomposition · p. 18

P Wrong-Diagnosis Control

Under the tested protocol, wrong same-type diagnoses do not measurably reduce remediation scores, including when diagnosis presentation is emphasized. Out-of-family tutor results and reproducibility details provide additional checks.

P Wrong-Diagnosis Control

  • Across three judges, wrong-diagnosis RSR is 3.2–4.2 pp above gold; GPT-5-mini comparisons versus C2 and B1 are not statistically significant.The control supplies an intentionally mismatched same-type label, so its conditioning claim is limited to the tested diagnosis-presentation format.Source: Table 13 · p. 18, P Wrong-Diagnosis Control · p. 18
  • With diagnosis stated first and an explicit directive to address it, wrong- and gold-diagnosis RSR are 82.1% and 81.7%, respectively (∆=+0.4 pp; p=0.93).The paired bootstrap 95% confidence interval is [−3.8, +4.7] pp; the authors scope the conclusion to the tested simulator and tutors.Source: P Wrong-Diagnosis Control · pp. 18–19, Table 15 · p. 19

Q Out-of-Family Tutor Replication Details

R Reproducibility

  • Reproducibility details report Python 3.11, Transformers 5.3.0, PyTorch 2.6.0, and NVIDIA H100 80GB GPUs for local model experiments.Phi-4 uses dual-GPU inference when tutor and evaluator are co-loaded.Source: R Reproducibility · p. 19
Loading 2610.31679v1…