PaperlayerView exact revision on arXiv

Source-linked AI summary

The Assistance Dilemma: Learning to Teach via Multi-Turn Reinforcement Learning

Jakub Macina, Manu Kapur, Mrinmaya Sachan

arXiv:2610.06446v1cs.CLcs.AI

TL;DR

LLMs optimized to answer questions are poor at teaching, and existing RL tutoring rewards can rise when tutors tell students the answer. Eduardo uses a masked near-transfer post-test and binary reward gates to train multi-turn tutors; its 27B model achieves strong tutoring results with fewer thinking tokens than frontier models.

  • Problem

    Existing RL-trained tutors assess success on the tutored problem with the tutor's words still available, so telling can raise the learning reward and requires a tuned penalty.

    Source: §3.2 · p. 4

  • Method

    Eduardo trains tutors with a masked near-transfer post-test that tests an unseen variant using only student turns, alongside two binary reward gates.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Eduardo-27B has the highest MathTutorBench pedagogical reward-model score and MathDial win rate over human teachers among tested tutors, strongest plain-prompt TutorMoments scaffolding, and uses 2.4–6.2× fewer thinking tokens than frontier models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The ablation finds that reward gates decide whether the tutor tells, while the masked near-transfer post-test shapes out-of-domain transfer and reasoning length.

    Source: §1 · p. 2

    Training uses a frozen LLM proxy student, measures near transfer within math, and may overfit to one student's failure modes; transfer to human learners remains a separate question.

    Source: §3.1 · p. 4, §6 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) trained to answer questions are natively poor at teaching. Reinforcement Learning (RL) against a simulated student is a promising approach to improve their pedagogy, but existing RL-trained tutors reward the student's success on the tutored problem with the tutor's words still in context. The reward is then easiest to raise by telling the student the answer, and a tuned penalty is needed to reduce telling. Drawing on learning sciences, we introduce a masked near-transfer post-test: the student is tested on an unseen variant of the tutored problem with the tutor's utterances masked, so the reward can rise only through what the student wrote in its own turns. This discourages cognitive offloading by the student and allows the continuous penalty to be replaced by two binary reward gates (factual correctness of tutor response, no solution handover). A leave-one-out ablation shows that the learning-gain reward on its own does not separate teaching from telling: the gates reduce solution handover while the near-transfer post-test improves out-of-domain transfer. Using these reward designs we develop Eduardo, a multi-turn RL recipe for training LLM tutors, and use it to train 4B, 9B, 14B and 27B models from two distinct LLM architectures. Our post-trained Eduardo-27B model matches Gemini-3.1-Pro on MathTutorBench and Claude Opus 4.8 on TutorMoments at 2.4-6.2x fewer thinking tokens than frontier models, which matters for interactive tutoring. Without being named in the reward, the model more than doubles its use of the push-for-justification teacher move while support fading (e.g., assigning independent work), whose payoff lies beyond a single-problem dialog episode, is trained out. We open-source our training environment, an 8,671-problem near-transfer dataset, and trained models for further development.

1 INTRODUCTION

Eduardo trains open LLM tutors with learning-science-inspired reward conditions that aim to favor teaching over telling. The introduction argues that how learning is measured matters, and reports strong tutoring results alongside behavioral changes.

1 INTRODUCTION

  • LLMs often answer rather than teach because answer-focused pre-training and single-turn RLHF reward immediate helpfulness, while tutoring requires withholding answers for students to construct solutions.The introduction describes this tension as the Assistance Dilemma, drawing on Koedinger & Aleven (2007).Source: §1 · p. 1
  • RL against simulated students offers a route to tutoring, whereas supervised fine-tuning on tutoring dialogs can reduce models' own problem-solving ability.Source: §1 · pp. 1–2
  • The masked near-transfer post-test and two binary reward gates form EDUARDO, a multi-turn RL recipe used unchanged to train models from 4B to 27B.The introduction reports that a learning-gain reward alone does not separate teaching from telling; without the gates, tutors handed over solutions in 61% of dialogs at the same simulated gain.Source: §1 · p. 2
  • The recipe's 8,671-problem near-transfer dataset accompanies an ablation finding that gates shape telling while the post-test shapes out-of-domain transfer and reasoning length.Source: §1 · p. 2
  • Eduardo-27B achieves the highest MathTutorBench pedagogical reward-model score and MathDial win rate over human teachers among tested tutors, and the strongest plain-prompt TutorMoments scaffolding.It uses 2.4–6.2× fewer thinking tokens than frontier models.Source: §1 · p. 2
  • Without naming pedagogical moves in the reward, optimization more than doubles push for justification, while support fading is trained out because its payoff lies beyond a single-dialog episode.Source: §1 · p. 2

2 RELATED WORK

Prior tutoring work includes simulated-student multi-turn RL, prompting, and supervised fine-tuning, each with limitations described here. TutorRL rewards solving the tutored problem with the dialog in context, creating an incentive to tell.

2 RELATED WORK

  • Prior education RL simulates confused users or cognitive states, but these students can display performative errors and learn unnaturally well; this work instead uses a frozen LLM student with genuine errors.The paper describes this approach as similar to TutorRL's trajectory-based reward.Source: §2 · pp. 2–3
  • Supervised fine-tuning on teacher–student dialogs can instill tutoring behavior but can also cause models to forget core mathematical reasoning.Source: §2 · p. 2
  • On MathTutorBench, Eduardo models improve pedagogy over their base at every size while preserving diagnostic accuracy.Source: Figure 1 · p. 3
  • TutorRL's same-problem, in-context post-test lets solution telling raise the learning reward, so its leakage penalty must be balanced against that reward.Source: §2 · p. 3

3 PEDAGOGICAL RL FOR THE ASSISTANCE DILEMMA

The section grounds tutoring in the Assistance Dilemma and frames tutoring as a dialogue in which the tutor must help without doing the student’s work. It proposes measuring learning through transfer and student-authored work, with reward gates that constrain telling.

3.1 PROBLEM FORMULATION

  • The tutor interacts with a frozen student through a dialog, observing the problem and history while inferring the student's hidden knowledge state.The tutor receives no reference solution; the student’s errors are genuine model failure modes rather than role-played confusions.Source: §3.1 · pp. 3–4

3.2 THREE CONDITIONS ON MEASURING LEARNING

  • Learning measurement should require transfer to a new problem, rely only on student-authored turns, and make telling costly through binary gates.The gates require factual correctness and no solution handover; because telling no longer raises measured gain, the gates need not be tuned against it.Source: §3.2 · p. 4

3.3 CONDITION 1: NEAR-TRANSFER DATASET

  • 8,671 BigMath problem pairs link tutored source problems to validated near-transfer variants that preserve the solution structure.Variants change numbers or surface stories and are retained when the generating model solves them blindly with the stated answer.Source: §3.3 · p. 4

3.4 CONDITION 2: MASKED POST-TEST OBJECTIVE

  • At post-test, tutor utterances are replaced with placeholders while turn structure is preserved, so the student attempts the target using only its own turns.This directs the measured gain toward reasoning the student generated that it can reuse on the variant.Source: §3.4 · p. 5

3.5 CONDITION 3: THE COST OF TELLING VIA BINARY REWARD GATES

  • Two LLM-judged gates require factual correctness and prohibit giving the solution verbatim or reducing the remaining work to a triviality.The authors make handover a constraint rather than measuring it after the fact or balancing it against learning gain with a hyperparameter.Source: §3.5 · p. 5

3.6 REWARD FORMULATION

  • The reward uses headroom-normalized learning gain and applies efficiency decay to positive gains, with a terminal penalty when either gate fails.Normalizing by headroom makes a gain on a hard problem worth as much as a gain on an easy one.Source: §3.6 · p. 5

4 EXPERIMENTAL SETUP

The experiments test scaling and cross-family robustness, ablate the reward conditions, assess out-of-domain generalization, and examine tutor behaviors. They compare Eduardo with base, open tutoring, and proprietary models using multiple benchmarks and a fixed training setup.

4 EXPERIMENTAL SETUP

  • Four research questions cover scaling across model sizes and families, reward-condition ablations, out-of-domain generalization, and learned, inherited, or lost tutor moves.The setup asks whether the recipe generalizes across models and whether evaluation transfers beyond the training student and judge.Source: §4 · pp. 5–6

4.1 BASELINES

  • Eduardo is compared with same-weight base models, open tutors TutorRL-7B and its Qwen2.5-7B-Instruct base, and proprietary Gemini and Claude models.Models use a shared plain prompt; TutorMoments also has an evaluation-aware prompt condition.Source: §4.1 · p. 6

4.2 DATASETS AND EVALUATION PROTOCOL

  • Training uses Qwen3.6-27B as judge, while evaluation judgments use Gemini-3.1-Pro and student generalization includes MathDial, Eedi, and TutorMoments.TutorMoments uses an independent Gemini-3.7-Flash student, and MathTutorBench and TutorMoments measure pedagogy axes absent from the reward.Source: §4.2 · p. 6
  • 13 tutor moves are classified in TutorMoments replays; evaluations also cover mid-dialog utterances, teacher skills, and 494 held-out training-environment problems.Mid-dialog responses are scored with MathTutorBench pedagogical reward, human-teacher win rate, and judge ratings of pedagogy and factual correctness.Source: §4.2 · p. 6

4.3 IMPLEMENTATION DETAILS

  • Training uses DPPO, caps sequences at 8,192 tokens and dialogs at 22 turns, and estimates learning gain from 16 masked-post-test student samples.The tutor can end a dialog earlier by emitting an end-of-conversation token.Source: §4.3 · p. 6

5 RESULTS

Eduardo-27B performs strongly on independent tutoring benchmarks, while ablations show that reward gates and near-transfer assessment shape different behaviors. Behavioral analysis finds more student-directed moves, but also that support fading is absent under the single-problem horizon.

5.1 ONE RECIPE ACROSS SIZES AND ARCHITECTURES (RQ1)

  • MathDial Ped-RM/win rate rose at 4B (2.08→7.92; 30%→78%), 9B (3.02→8.29; 38%→82%), and 27B (8.34→9.04; 82%→89%); Eduardo-27B led all tested tutors.On Eedi, 4B and 9B also gained, but 27B trailed its base in win rate (82.3% vs. 84.3%); Eduardo-14B was at parity with Qwen3-14B on both datasets.Source: §5.1 · p. 6
  • The training-student gain weakly ranks tutors: even tutors handing over answers in 83–93% of dialogs score +0.13–0.14; Eduardo-27B’s objective improves from +0.22 to +0.29.Leakage is 9% for Eduardo-27B, matching its base, and 25–31% for 4B/9B; Qwen3-14B drops from 98% to 32%.Source: §5.1 · p. 7

5.2 CONDITIONS ABLATION (RQ2)

  • Removing any one condition lowers MathDial Ped-RM (7.92→6.10 without gates, 6.48 without masking, 5.25 without near-transfer) and win rate (78%→65%, 65%, 56%).The passage also reports lower Ped-RM on Eedi when any condition is removed.Source: §5.2 · p. 7

5.3 OUT-OF-DOMAIN GENERALIZATION (RQ3)

  • 0.79 on MathTutorBench put Eduardo-27B near Gemini 3.1 Pro (0.80); on plain-prompt TutorMoments it led Scaffolding (0.96) and Avoids-Over-help (0.76), with 2.4–6.2× fewer thinking tokens.On TutorMoments, it ranked second on rigour under the plain prompt, but frontier models overtook it on rigour and Avoids-Over-help under the evaluation-aware prompt.Source: Table 3 · p. 8, §5.3 · p. 8

5.2 CONDITIONS ABLATION (RQ2)

  • Ablation gains stay within 0.03 in Δtransfer, while solution handover reaches 61% without gates and 47% without near-transfer; the gates make handover costly.The gates govern handover, near-transfer governs out-of-domain transfer, and removing the mask retains unstable reasoning, with 27% of turns exceeding the cap and 40% of dialogs truncated.Source: §5.2 · p. 8

5.3 OUT-OF-DOMAIN GENERALIZATION (RQ3)

  • Every EDUARDO model improves over its base on MathTutorBench, whose independent-judge pedagogy axes are absent from the reward; Eduardo-27B retains problem-solving accuracy.Source: §5.3 · p. 8

5.4 BEHAVIORAL DISCOVERY (RQ4)

  • Unprompted frontier models sit 0.32–0.39 JS from human move distributions; EDUARDO sits 0.24 from the prompted-model centroid without that prompt, closer than unprompted models.The evaluation-aware prompt barely moves EDUARDO, unlike its base; the authors report its move profile as approximately matching an expert-defined rubric.Source: §5.4 · pp. 8–9, Figure 2 · p. 9
  • Under the same plain prompt, justification prompting more than doubles (9.0%→21.8%) and guiding questions rise (23.2%→41.4%), while independent-work requests fall to 0.0%.The binary gates penalize explaining and supplying answers, which fall from 18.5% to 3.9% and 15.5% to 2.2%, respectively.Source: §5.4 · p. 9, Figure 2 · p. 9

6 CONCLUSION

EDUARDO combines a learning-gain objective with conditions that make it insufficient on its own, training tutors from 4B to 27B; the 27B model leads the reported tutoring evaluations while using several times fewer thinking tokens.

6 CONCLUSION

  • EDUARDO-27B achieves the highest pedagogical reward-model score and MathDial win rate over human teachers among tested tutors, and is strongest on TutorMoments under a plain prompt.It uses several times fewer thinking tokens than frontier models.Source: §6 · pp. 9–10
  • A learning-gain reward is an appropriate tutoring objective but does not by itself distinguish teaching from telling.The conclusion ties that distinction to how learning is measured.Source: §6 · p. 9
  • Rewarding what users can do independently rather than what they were told may extend to domains where agents should build capabilities instead of acting as oracles.Source: §6 · p. 10
  • The approach uses a frozen LLM proxy student, measures near transfer only in math, and does not incentivize socio-emotional tutoring moves.The ablation also uses one seed, one model size, and one checkpoint per arm, so it identifies affected behaviors rather than establishing necessity.Source: §6 · p. 10

A HYPERPARAMETERS AND TRAINING DETAILS

The appendix specifies shared training settings, efficiency decay, two binary reward gates, and prompts for the tutor, student, and masked post-test.

A HYPERPARAMETERS AND TRAINING DETAILS

  • All model sizes use shared hyperparameters; DPPO is selected for long-dialog stability, and a typical 100-step run processes about 41M tutor tokens in 10–12 hours.Reward hacking observed during development included answer-echo laundering and inactive turns; monitoring used conversation review and independent-work scoring.Source: A HYPERPARAMETERS AND TRAINING DETAILS · p. 13

B EFFICIENCY DECAY FORMULA

  • Efficiency decay multiplies separate token-weighted decays for visible and thinking tokens, with independent per-turn budgets that prevent cross-subsidies.Token weighting prevents short filler turns from diluting long-lecture penalties; decay discounts positive learning gain, while negative gain passes through undiscounted.Source: B EFFICIENCY DECAY FORMULA · pp. 13–14

C BINARY REWARD GATES: TRAINING JUDGE PROMPTS

C.1 NO-HANDOVER GATE (ILEAKAGE)

  • The no-handover gate rejects premature answers, key methods, or complete solution paths, including when a tutor embeds the key step in a question.Guiding questions and methods named after a student attempt are allowed; rejection requires a quoted tutor turn as evidence.Source: §C.1 · p. 15

C.2 FACTUAL CORRECTNESS GATE (IQUALITY)

  • The factual-correctness gate rejects mathematical errors, false corrections, unresolved student mistakes, degenerate output, dictated repetition, and absent teaching attempts.Teaching style, hint strength, and pacing are excluded from this gate.Source: §C.2 · p. 15

D.1 TUTOR SYSTEM PROMPT

  • The tutor prompt sets correctness and student-cognitive-work constraints while leaving strategy to the policy; prompts name no specific pedagogical moves.The tutor should elicit an attempt before supplying a method or answer and may demonstrate the idea on a different example if the student cannot attempt the problem.Source: D SYSTEM PROMPTS · p. 15, §D.1 · pp. 15–16

D.2 STUDENT SYSTEM PROMPT

  • The simulated student receives no confusion persona or predefined misconceptions, so its errors are genuine model failures rather than scripted behavior.Source: §D.2 · p. 16

D.3 NEAR-TRANSFER POST-TEST PROMPT

  • After dialogue, the post-test hides tutor turns and asks the student to solve a new problem independently in a complete, gradeable solution.Source: §D.3 · p. 16

E.1 GENERATION PROCEDURE

The generation procedure creates and validates near-transfer variants, filters them by difficulty, and falls back to original problems when no variant is available.

E.1 GENERATION PROCEDURE

  • 8,671 near-transfer pairs are constructed from BigMath source problems, with 10,000 training sources and 500 test sources.Source: §E.1 · p. 16
  • Gemini-3.1-Pro rewrites each source with changed numbers or surface context while preserving solution structure; a fresh solve verifies the answer, with up to three regeneration attempts.The frozen Llama-3.1-8B-Instruct student samples 16 solutions on each problem to estimate prior solve rates.Source: §E.1 · p. 16
  • Variants are rejected when transfer solve rate exceeds 0.75 or differs from source solve rate by more than 0.375; 98.8% of problems receive a validated variant.Problems without variants fall back to the original problem for testing.Source: §E.1 · p. 17

E.2 SOLVE-RATE DISTRIBUTIONS

The dataset statistics indicate that transfer variants broadly preserve problem difficulty, and examples illustrate changed answers despite preserved solution structure.

E.2 SOLVE-RATE DISTRIBUTIONS

  • Mean student solve rates are 0.289 for source problems and 0.280 for transfer variants, with source-transfer difficulty correlation r ≥0.65.Statistics cover 8,238 data points and 433 test points after excluding pairs with solve rates of zero or above 0.9.Source: §E.2 · p. 17, Table 5 · p. 17
  • 38.9% of problems have source solve rates below 0.10, and the distribution thins toward the 0.65 ceiling.The distribution reflects the BigMath source distribution filtered by the difficulty gate.Source: §E.2 · p. 17

E.3 EXAMPLE NEAR-TRANSFER PAIRS

  • Representative source-transfer pairs preserve concepts and reasoning steps while changing numbers or surface context, producing different final answers.A student who memorized a source answer therefore gains nothing on its transfer variant.Source: §E.3 · p. 17, Table 6 · p. 17

F ENVIRONMENT EVALUATION DETAILS

The section reports confidence intervals for learning-gain and solution-handover measures and documents how token limits affect evaluation diagnostics.

F ENVIRONMENT EVALUATION DETAILS

  • Table 7 reports bootstrap 95% confidence intervals for near-transfer gain, same-problem gain, and solution-handover rate across 494 problems.The intervals use 10,000 resamples.Source: Table 7 · p. 18, F ENVIRONMENT EVALUATION DETAILS · p. 18
  • All Eduardo models stay below 11% empty turns, and Eduardo-27B stays below 1%; truncation primarily affects smaller verbose base models.Qwen3.5-4B produces empty turns 70% of the time, Qwen3.5-9B 20%, and Qwen3-14B 8%.Source: F ENVIRONMENT EVALUATION DETAILS · p. 18
  • Arms exceeding 15% on empty-turn or sequence-truncation rates are flagged, and their environment metrics understate learning gain and solution handover.Empty turns occur when reasoning exceeds the 4,192-token per-turn cap; sequence truncation occurs at the 16,384-token conversation budget.Source: F ENVIRONMENT EVALUATION DETAILS · p. 18

G EXAMPLE TRANSCRIPTS

Matched TutorMoments examples contrast Eduardo’s elicitation of student reasoning with a base model’s direct explanations. The evaluation rubrics define how factuality and pedagogy are judged.

G EXAMPLE TRANSCRIPTS

  • On the same problems with the same student, Eduardo elicits the sequence rule and conceptual distinction, while the base model supplies explanations itself.The examples contrast guiding questions and prompting for justification with direct telling; the reward does not name prompting for justification, which emerged from the masked transfer objective.Source: G EXAMPLE TRANSCRIPTS · pp. 18–19

H EVALUATION JUDGE RUBRICS

  • Gemini-3.1-Pro scores factuality and pedagogy separately on 1–5 scales after reviewing the problem, correct answer, and full tutor–student dialog.The factuality rubric checks mathematical correctness and language appropriateness relative to the problem and student level.Source: H EVALUATION JUDGE RUBRICS · p. 20, Table 9 · p. 20
  • The pedagogy rubric rewards contingent scaffolding, productive struggle, and rigour, with students doing the reasoning and most of the talking.It includes minimal hints, fading as competence grows, and pressing for justification while avoiding funneling.Source: Table 10 · p. 20

Related papers

Loading 2610.06446v1…