PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Response Framework for Questions and Tutoring Dialogues

Zhangqi Duan, Simon Woodhead, Maya Miller-Vedam, Andrew Lan

arXiv:2610.19532v1cs.CLcs.CYcs.LG

TL;DR

Existing student models largely focus on question responses, while tutoring dialogues may provide finer-grained evidence about student knowledge. URF-QD converts tutor-posed mathematical subtasks into standalone questions and models them with original questions on a shared IRT scale. On two real-world math education datasets, it improves both question difficulty prediction and response correctness prediction.

  • Problem

    Existing response models focus on question records, and there is no rigorous way to model original questions and tutor-posed dialogue subtasks together, despite dialogues offering finer-grained evidence about student knowledge.

    Source: §1 · pp. 1–2

  • Method

    URF-QD rewrites tutor-posed mathematical subtasks as standalone questions and jointly models them with original questions using dynamic IRT and tutor-move relative-order regularization.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    URF-QD improves both question difficulty prediction and response correctness prediction across two real-world math learning datasets.

    Source: §5.1 · pp. 6–7, §7 · p. 8

  • Takeaways & Limitations

    The framework uses dialogue subtasks and pedagogical relations to estimate student knowledge and task difficulty on a shared scale.

    Source: §7 · p. 8

    Evaluation is constrained by limited public availability of the two datasets, and reference difficulty labels are IRT estimates that may not reflect true difficulty.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Existing student models in educational applications largely focus on student responses to questions. However, as tutoring becomes more accessible at large scale, it is important to analyze dialogues between tutors and students, which may reveal fine-grained evidence on student knowledge. In this work, we propose a unified framework that models both student responses to questions and tutor-posed subtasks during dialogues on a shared item response theory (IRT) scale. We first identify mathematical subtasks in tutoring dialogues, rewrite them into self-contained questions, and identify the correctness of student responses to these subtasks. We then develop an IRT framework that estimates question/subtask difficulty from text, tracks students' evolving abilities across question-answering and dialogue interactions, and a novel method to leverages tutor moves between consecutive subtasks to regularize their difficulty values. We evaluate our framework on two real-world math education datasets containing both human- and AI-tutored dialogues. Results show that our framework outperforms strong baselines on both question difficulty prediction and response correctness prediction.

1 Introduction

Response-only IRT and knowledge-tracing models do not analyze original question responses and tutor-posed subtasks together, although dialogues may reveal finer-grained evidence about student knowledge. URF-QD models both on a shared IRT scale by converting subtasks into standalone questions and using tutor moves to regularize their relative difficulty.

Motivation

  • Tutoring dialogues contain smaller assessment and learning events, including recall, calculations, and reasoning, that may reveal changing knowledge beyond an original answer.The example includes an initial conceptual error, success on a simpler scaffolded task, correct intermediate calculations, and a remaining justification gap.Source: §1 · pp. 1–2

Motivation and gap

  • Response-only IRT and knowledge-tracing models do not provide a rigorous way to analyze original questions and dialogue subtasks together.Personalized dialogue trajectories make subtasks difficult to calibrate with traditional IRT, and context-dependent subtasks can be difficult to represent as standalone questions.Source: §1 · pp. 1–2

Contributions

  • URF-QD jointly models original-question responses and mathematical dialogue subtasks on a shared IRT scale.It identifies tutor turns posing mathematical tasks, rewrites context-dependent tasks as self-contained questions, and introduces tutor-move regularization of relative difficulty.Source: §1 · p. 2
  • The framework is evaluated on two real-world math education datasets, and the reported evaluation shows improvements in question difficulty and response correctness prediction.The authors also report human evaluation and qualitative analyses of framework components.Source: §1 · p. 2

2 Problem Formulation

The formulation treats original questions and tutor-posed mathematical subtasks as scored interactions in a chronological student history. Its objective is to predict response correctness using that history while placing tasks and student ability on a common IRT scale.

Problem Formulation

  • A dialogue subtask is a tutor-posed task requiring a mathematically meaningful student response, such as a calculation, recall, method choice, interpretation, or explanation.Student responses to these tasks are treated as additional response observations, and the tasks are transformed into standalone questions.Source: §2 · p. 2
  • URF-QD represents original questions and rewritten dialogue subtasks on a shared IRT scale, with evolving student ability and tutor-move relative-order regularization.The framework overview distinguishes the response-correctness objective from the question-difficulty objective.Source: Figure 2 · p. 3
  • Each student's scored interactions are ordered chronologically as task-response pairs, where each pair may represent an original question or a dialogue subtask.The response correctness value is binary.Source: §2 · pp. 2–3
  • The objective is to estimate the probability of a correct response from the current task and preceding interaction history while placing ability and task difficulties on a common IRT scale.Question-answering and tutoring-dialogue interactions provide complementary observations for modeling student performance.Source: §2 · p. 3

3 Methodology

URF-QD places original questions and tutor-posed dialogue subtasks on a shared IRT scale. It transforms subtasks into standalone questions, tracks evolving student ability, and uses tutor moves to regularize relative task difficulty.

3 Methodology

  • The framework jointly models original-question and dialogue-subtask responses on a shared IRT scale, with text-based difficulty estimates and evolving student ability.Dialogue subtask responses also update the student state and refine future ability estimates.Source: §3 · p. 3, §3.2 · p. 4

3.1 Subtask Transformation

  • Tutor turns followed by student responses are screened for explicit mathematical tasks and attempted answers; retained subtasks are rewritten as self-contained questions and mapped into students’ chronological interaction histories.Turns without an applicable mathematical response are left unscored.Source: §3.1 · p. 4

3.2 URF-QD: Joint Modeling of Questions and Dialogue Subtasks

  • A shared task encoder maps question text to difficulty, while a dynamic student state is updated from task representations and response correctness to estimate preinteraction ability.The same difficulty head is used for original questions and subtasks, allowing text-based difficulty estimates to generalize across tasks, students, and dialogues.Source: §3.2 · pp. 4–5

3.3 Difficulty Order Regularization

  • Tutor moves provide soft relative-difficulty constraints: scaffolding after struggle implies an easier next subtask, while pushing for rigor after a correct response implies a harder one.No constraint is assigned when correctness is unscored, subtasks are unrelated, or cognitive demand does not substantively change.Source: §3.3 · p. 5

3.4 Prediction Targets, Training Objectives

  • Difficulty prediction combines MSE on fitted IRT difficulties for original questions, BCE on subtask response correctness, and the ordering penalty; response prediction combines BCE with that penalty.Dialogue responses also update the knowledge state used in future correctness predictions.Source: §3.4 · pp. 5–6

4 Experimental Evaluation

The evaluation uses two math-education datasets with human- and AI-tutored dialogues, assessing question difficulty and response correctness prediction against specified baselines under five-fold cross-validation.

4.1 Datasets

  • The evaluation includes Eedi human-tutored dialogues and Khan Academy AI-tutored dialogues, with 588 and 136 students, respectively.The processed Eedi data has 3,689 questions, 1,775 dialogues, and 152,280 question-answering records; Khan Academy has 3,423 questions, 1,800 dialogues, and 56,529 records.Source: §4.1 · p. 6

4.2 Metrics

  • Difficulty prediction is evaluated with PCC, SCC, MSE, and MAE, while response correctness prediction uses AUC, accuracy, and F1.Source: §4.2 · p. 6

4.3 Baselines

  • Difficulty baselines are kNN Mean, Random Forest, ICL kNN, and Diff SFT; correctness baselines include DKT, DKT (Joint), AKT, simpleKT, and URF-QD (Orig. Question).Joint denotes training on original-question and dialogue-subtask responses; Reg. denotes relative difficulty order regularization.Source: §4.3 · p. 6

4.4 Experimental Setup

  • Performance is averaged over five-fold cross-validation, splitting by problems for difficulty prediction and by students for correctness prediction to test generalization to unseen problems or students.Source: §4.4 · p. 6

5 Results, Analysis, and Discussion

The section reports quantitative results for difficulty and correctness prediction, followed by an example of how dialogue-subtask supervision affects difficulty estimates.

5.1 Quantitative Results

  • URF-QD (Joint + Reg.) outperforms all baselines on four difficulty metrics; on Eedi versus the strongest baseline, it reduces MSE by 7.7% and improves PCC and SCC by 6.5% and 8.0%, respectively.The improvements are statistically significant for MSE, PCC, and SCC on Eedi and for PCC on Khan Academy (p < 0.05 under paired t-test).Source: §5.1 · pp. 6–7, Table 1 · p. 7
  • The Eedi ablation shows that dialogue subtasks improve both prediction tasks, while relative-order regularization adds gains for difficulty prediction.Source: §5.1 · p. 7
  • For correctness prediction, URF-QD achieves the best performance on every metric for both datasets; on Eedi, adding dialogue subtasks and relative-order regularization improves AUC by 6.3% and accuracy by 4.5% over URF-QD (Orig. Question).On Khan Academy, the full model also improves all three metrics over its original-question-only counterpart.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 Qualitative Results

  • In the mixed-number division example, URF-QD predicts difficulty 0.573, closer to ground truth 0.625 than Diff SFT’s 0.483 estimate.Relative-order regularization assigns difficulties of 0.487 and 0.256 to the preceding and scaffolded subtasks, respectively; URF-QD’s absolute error is 0.052 versus Diff SFT’s 0.142.Source: §5.2 · pp. 7–8, Table 3 · p. 8

6 Human Evaluation

Human evaluations on Eedi assessed tutor-move annotation reliability and the quality of rewritten subtasks.

6 Human Evaluation

  • GPT-5.4 achieves a macro-F1 of 0.754 for tutor-move annotation, and 85.5% of rewritten subtasks satisfy both completeness without dialogue context and preservation of mathematical demand.Source: §6 · p. 8

7 Conclusions and Future Work

The conclusion presents URF-QD as jointly modeling original questions and tutor-posed subtasks on a shared IRT scale. It identifies tutor analysis, adaptive subtask posing, richer question inputs, and broader educational settings as future directions.

7 Conclusions and Future Work

  • URF-QD jointly models original questions and tutor-posed subtasks on a shared IRT scale, transforming dialogue subtasks into standalone questions and using pedagogical relations to regularize difficulty estimates.Source: §7 · p. 8
  • The framework’s shared dynamic scale could support analysis of LLM tutor strategies and behavior, while future work could use extracted insights to guide adaptive subtask posing by human and LLM tutors.Source: §7 · p. 8
  • Future work could incorporate images and diagrams in math questions and examine generalization to other educational domains and settings.Source: §7 · p. 8

Limitations

The authors identify limits in dataset availability, reference difficulty labels, and the response and IRT model scope.

Limitations

  • Evaluation uses two real-world math datasets, but Khan Academy is not public and only a subset of Eedi is publicly accessible, constraining broader evaluation.Source: Limitations · p. 9
  • Because neither dataset has ground-truth question difficulty labels, reference labels come from fitted IRT estimates that may not reflect true difficulty.Source: Limitations · p. 9
  • The framework is restricted to 1PL IRT and binary correctness labels, which may not fully capture student knowledge and mathematical responses.Source: Limitations · p. 9

Ethical Considerations

The paper considers how to model question responses and dialogue subtasks together, while noting risks from inaccurate estimates and LLM-based annotations. Its related-work discussion positions URF-QD as a shared-scale approach that uses dialogue subtasks as additional response observations.

Ethical Considerations

  • Inaccurate ability or difficulty estimates and errors or biases in LLM-based dialogue annotations could affect recommendations and assessments; demographic data are unavailable for disparity evaluation.The authors say the framework is intended to support, not replace, educators’ judgments, and call for further fairness and outcome evaluation before deployment.Source: Ethical Considerations · p. 9

A.1 Question Difficulty Prediction and Student Response Modeling

  • Unlike item-centric difficulty methods, URF-QD uses real student responses to tutor-posed subtasks as additional supervision for predicting difficulty, including for unseen original questions..Source: §A.1 · p. 11
  • URF-QD adds dialogue-derived subtask responses to a shared IRT scale and tracks student ability across questions and tutoring interactions.The related-work discussion distinguishes this from methods centered on predefined exercise-response sequences or dialogue-response prediction.Source: §A.1 · p. 11, §A.2 · p. 11

A.2 Student Modeling in Tutoring Dialogues

  • Prior dialogue approaches address response prediction, while URF-QD jointly models original-question and dialogue-subtask difficulties to predict unseen-question difficulty.Source: §A.2 · p. 11

B Additional Dataset Details

  • The additional dataset details state that Eedi and Khan Academy records are anonymized and associate dialogue knowledge components with those of the linked question.The supplied passages report 4,175 Eedi subtask instances across 1,775 dialogues and 4,996 Khan Academy subtask instances across 1,800 dialogues.Source: B Additional Dataset Details · p. 11

C.1 Baselines

  • The additional experimental details cover baseline methods and implementation; difficulty-prediction baselines use original questions and reference difficulties without dialogue-subtask response supervision or ordering regularization.The listed baselines include fine-tuned language models, random forest, kNN Mean, and ICL kNN; response-prediction comparisons include DKT, AKT, and simpleKT.Source: §C.1 · p. 12

C.2 Implementation Details

The implementation uses frozen language encoders for question representations and a sequential student-state model for response prediction. Reference abilities and difficulties come from a fitted 1PL IRT model.

C.2 Implementation Details

  • GPT-5.4 supports subtask identification, correctness labeling, standalone-question transformation, and pedagogical-move annotation; Qwen3.5-4B is the frozen framework encoder.The correctness annotation protocol is reported to have Cohen’s κ=0.722 agreement with experts, compared with human interannotator κ=0.719.Source: §C.2 · p. 12
  • A three-layer MLP predicts scalar question difficulty from 2560-dimensional frozen-encoder embeddings, trained with AdamW for 120 epochs.The implementation uses GELU activations, full-batch optimization, learning rate 5 · 10^-3, and gradient clipping at 10.0.Source: §C.2 · p. 12
  • Reference student abilities and question difficulties are estimated by fitting a 1PL IRT model to eligible original-question responses without a train/validation/test split.The fitted responses include questions later assigned to validation and test sets; L-BFGS-B and L2 regularization are used.Source: §C.2 · pp. 12–13
  • An LSTM with 200-dimensional hidden state models evolving student state, which is projected to scalar ability before the current response is predicted.Correctness embeddings and question embeddings form the LSTM input; the response model is trained for 10 epochs, and both prediction tasks use λ = 1.Source: §C.2 · p. 13
  • The implementation computes AUC with scikit-learn and Pearson and Spearman correlations with NumPy.The authors say they will ensure released code reflects the licenses and terms of the resources used.Source: §C.2 · p. 13

D.1 Additional Ablation Study

The Eedi ablations report that dialogue subtasks, relative-order regularization, rewriting, and the larger encoder contribute differently across prediction targets. Moderate ordering regularization improves pedagogical-order consistency, while excessive strength harms prediction performance.

D.1 Additional Ablation Study

  • Removing relative-order regularization consistently reduces question-difficulty performance, but does not hurt response-correctness prediction.The authors suggest sparse ordering constraints may be diluted by the larger number of interaction records in response prediction.Source: §D.1 · pp. 13–14
  • Using original tutor subtasks generally beats Diff SFT on MSE, PCC, and SCC, while standalone rewriting further improves all four difficulty metrics.The MAE comparison between original subtasks and Diff SFT is slightly worse for the original-subtask variant.Source: §D.1 · p. 13, Table 4 · p. 13
  • Replacing Qwen3.5-4B with MPNet-based SBERT raises MSE from 0.350 to 0.398 and lowers PCC from 0.458 to 0.341 in difficulty prediction.Source: §D.1 · p. 13, Table 4 · p. 13
  • Removing dialogue subtasks worsens question-difficulty and response-correctness prediction, indicating that subtasks provide information beyond original questions.For response prediction, AUC falls from 0.637 to 0.599 and accuracy from 0.598 to 0.572 in the original-question-only variant.Source: §D.1 · pp. 13–14, Table 4 · p. 13, Table 5 · p. 13
  • Jointly fine-tuning Qwen3.5-4B reduces performance on both tasks and substantially increases training time, favoring a frozen encoder in this setup.Difficulty MSE and MAE rise to 0.408 and 0.486; response AUC, accuracy, and F1 fall to 0.626, 0.589, and 0.552.Source: §D.1 · p. 14

D.2 Impact of Relative-Order Regularization

  • For difficulty prediction, performance improves as λ rises to 0.5 and 1, stays comparable at 3, and drops clearly at 10; response prediction is stable at moderate λ but worsens at 10.The authors attribute the high-λ losses to overemphasis on ordering relative to absolute difficulty fitting or response-prediction objectives.Source: §D.2 · p. 14
  • The overall ordering-satisfaction rate rises from 0.699 at λ = 0 to 0.899 at λ = 1, with gains for scaffolding and pushing-for-rigor pairs.At λ = 10 satisfaction reaches 0.914, but difficulty prediction worsens, showing a trade-off between ordering alignment and absolute difficulty estimation.Source: §D.2 · p. 14

E Human Evaluation Details

Human evaluation assesses tutor-move annotations and standalone subtask rewrites, while supplementary analyses relate task difficulty to later responses and compare human tutor tasks with GPT-5.4’s. The passages also describe potential benchmarking uses and the rewriting safeguards.

E Human Evaluation Details

  • GPT-5.4 achieves macro-averaged positive-class F1 of 0.754 for the two pedagogical moves, with F1 of 0.833 for scaffolding and 0.674 for pushing for rigor.Human annotator Fleiss’s κ is 0.651 for scaffolding and 0.591 for pushing for rigor; the latter move is less frequent, with 48 versus 467 labeled pairs, which may partly contribute to lower agreement and F1.Source: E Human Evaluation Details · p. 15
  • Annotators judged 92.7% of rewritten subtasks self-contained and solvable, 90.0% to preserve mathematical demand, and 85.5% positive on both criteria.The evaluation covered 110 rewritten subtasks.Source: E Human Evaluation Details · p. 15

F LLM Tutor Benchmarking

  • The preliminary LLM-tutor benchmark discussion proposes evaluating whether tutors pose scaffolding tasks at difficulty levels associated with student learning outcomes.It contrasts this direction with benchmarks described as relying on LLM-as-a-judge rather than principled psychometric models.Source: F LLM Tutor Benchmarking · pp. 15–16

Task difficulty and next question correctness

  • Across 553 dialogues and 514 subsequent responses, larger mean student-ability-minus-subtask-difficulty gaps are associated with higher next-question correctness (slope 0.613, p = 0.001).The analysis uses a five-fold student-disjoint dynamic 1PL model and filters for a subsequent original question sharing at least one knowledge component.Source: Task difficulty and next question… · p. 16, Figure 4 · p. 16
  • In 2,994 paired task comparisons across 916 dialogues, all but one of 21 human tutors posed easier subtasks on average than GPT-5.4; the reported human-tutor advantage is 1.11 percentage points (p = 0.006).The authors leave comparisons across other LLMs and pedagogical prompts to future work.Source: Task difficulty and next question… · p. 16, Figure 5 · p. 16
  • The proposed benchmark use links tutor-posed task difficulty with learning outcomes and could support analysis of tutor strategies and adaptive-testing trade-offs.The authors describe an outcome-based tutoring benchmark relating LLM tutoring behavior to student learning outcomes.Source: Task difficulty and next question… · p. 16

G.2 Standalone-Question Transformation

  • For standalone rewriting, the prompt uses preceding dialogue to resolve references and recover minimal context while preserving the tutor’s mathematical demand.The following student response is sealed reference context and cannot supply answer-specific information to the rewrite.Source: §G.2 · p. 17

Related papers

Loading 2610.19532v1…