PaperlayerView exact revision on arXiv

Source-linked AI summary

Sherpa: Teaching LLMs to Teach Adaptively

Weixian Xu, Yanzhe Zhang, Zora Zhiruo Wang, Changyu Chen, Diyi Yang

arXiv:2610.08778v1cs.AIcs.CL

TL;DR

LLM problem-solving ability does not guarantee effective teaching, and existing teacher-training signals are not grounded in diverse students’ learning outcomes. SHERPA trains a teacher through multi-turn interaction with student archetypes, optimizing for their improvement. It improves student performance across archetypes and reports gains on MathTutorBench and in human preference judgments.

  • Problem

    Existing LLM teacher-training signals rely on demonstrations, preference data, or predefined pedagogical criteria rather than diverse students’ learning outcomes, despite variation in students’ needs.

    Source: §1 · pp. 1–2

  • Method

    SHERPA trains a teacher with multi-turn reinforcement learning using student archetypes and rewards tutoring according to student improvement.

    Source: §1 · p. 2

  • Results

    20.5 percentage points: average student performance improves across archetypes; MathTutorBench pedagogy score rises from 52.5% to 79.2%, and the trained teacher wins 79.6% of human pairwise comparisons.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 7

  • Takeaways & Limitations

    SHERPA’s outcome-based training improves student outcomes across archetypes, with gains also reported on an external teaching benchmark and in human judgments.

    Source: §5 · p. 9

    Because the student backbone is frozen, the unaided baseline is independent of teacher parameters, making improvement maximization equivalent to maximizing post-tutoring success.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, preference data, or predefined pedagogical criteria that specify what good teaching looks like. However, these signals are often not grounded in individual student learning outcomes, where effective teaching strategies can vary substantially across learners. To address this, we introduce Sherpa, a multi-turn reinforcement learning framework that instantiates multiple student archetypes with LLMs conditioned on distinct learning preferences and trains a teacher model to adapt its instruction by directly maximizing their learning outcomes. Teacher LLMs trained with Sherpa improve instructed students' performance across all archetypes by an average of 20.5 percentage points. Under MathTutorBench's evaluation, Sherpa raises the overall pedagogy score from 52.5% to 79.2%, indicating better teaching responses. Our human studies show that the trained teacher is preferred over the base model in 79.6% of pairwise comparisons. Together, Sherpa trains LLM teachers to adapt to diverse simulated students and become better aligned with human teachers, paving the road towards AI tutors teaching real students.

1 INTRODUCTION

SHERPA trains a teacher through interaction with diverse student archetypes, optimizing for student improvement rather than predefined teaching criteria. It addresses the gap between LLMs’ problem-solving ability and effective teaching, with gains reported across student archetypes, an external benchmark, and human comparisons.

Motivation

  • LLMs’ strong problem-solving ability does not ensure effective teaching, and task assistance may not translate into learning or independent performance.Prior evaluations found none of 16 frontier models scored above 56% against expert-written tutoring rubrics.Source: §1 · p. 1

Research gap

  • Existing teacher-training signals rely on demonstrations, preference data, or predefined pedagogical criteria rather than diverse students’ learning outcomes.Students differ in preferences and needs, so effective teaching strategies vary across learners.Source: §1 · pp. 1–2

Approach

  • SHERPA uses multi-turn reinforcement learning with student archetypes and rewards tutoring according to student improvement, prompting teachers to adapt to individual needs.Its preparation–tutoring–testing cycle gives the teacher feedback based on the student’s improvement at test time.Source: §1 · p. 2

Reported results

  • 20.5 percentage points: SHERPA improves average student performance across archetypes; MathTutorBench pedagogy score rises from 52.5% to 79.2%, and human teachers prefer the trained teacher in 79.6% of pairwise comparisons.The reported student gains include archetypes unseen during training, and the MathTutorBench improvement is reported on an external benchmark.Source: §1 · p. 2

2 HOW TO TRAIN AN ADAPTIVE TEACHER LLM?

SHERPA trains an adaptive teacher through preparation, tutoring, and testing with student archetypes. Student improvement supplies the objective, while gates make student preferences consequential and inform turn-level policy updates.

2.1 PROBLEM FORMULATION

  • The teacher does not observe a student’s latent archetype and must infer instructional needs from replies while maximizing improvement on the assigned problem.Archetypes represent instructional preferences, with more nuanced preferences treated as combinations of representative patterns.Source: §2.1 · pp. 2–3

2.2 STUDENT ARCHETYPE

  • A weak student backbone makes tutoring gains easier to attribute, while gates help elicit distinct behaviors that prompting alone may not reliably produce.Archetype preferences include causal justification, subgoal decomposition, step demonstration, independent verification, contrastive comparison, and attempt diagnosis.Source: §2.2 · pp. 3–4
  • Student archetypes use guidance and adaptive gates: instructions that disclose the final answer are blocked, and preference-incompatible guidance is rejected with feedback.Only accepted exchanges reach the student backbone; rejection reasons remain available in the teacher context for later turns.Source: §2.2 · p. 4

2.3 TRAINING THE TEACHER POLICY

  • The teacher privately prepares a solution, tutors the student over multiple turns, then tests the student; reward is based on improvement over unaided performance.Repeated tests estimate performance with and without the tutoring interaction history.Source: §2.3 · pp. 4–5
  • Training masks rejected turns because they do not contribute to student improvement, then normalizes turn-level returns using leave-one-out comparisons within trajectory groups.Local format and repetition penalties are added after normalization; when only one trajectory reaches a turn, the comparison term is set to zero.Source: §2.3 · p. 5

3 ADAPTING TEACHING TO INDIVIDUALS

SHERPA trains teachers against diverse student preferences and evaluates their effects on simulated students, an external tutoring benchmark, and human judgments. Results favor diverse-preference training, while archetype comparisons and benchmark outcomes also reveal scope and trade-offs.

3.1 TEACHING DIVERSE ARCHETYPES

  • 69.0% overall test accuracy: SHERPADP outperforms the untrained backbone, PedagogicalRL, and SHERPANP on every student archetype.Training uses four archetypes; three others are held out for out-of-distribution evaluation.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2 EVALUATION BEYOND THE STUDENT ARCHETYPE

  • 79.2% Ped. Avg. on MathTutorBench exceeds the untrained backbone’s 52.5% and PedagogicalRL’s 63.2%, with gains on all four response-generation metrics.Mistake-location accuracy declines across trained teachers, although SHERPADP remains above PedagogicalRL at 30.3% versus 18.7%.Source: Table 2 · p. 6, §3.2 · pp. 6–7

3.3 HUMAN PAIRWISE COMPARISON

  • 79.6% of human pairwise judgments favored SHERPADP over the base model, excluding ties.Preference held across all four instructional preferences, including those unseen in training; some participants found SHERPADP’s explanations potentially overwhelming.Source: Table 3 · p. 7, §3.3 · p. 7

3.4 EVALUATING THE TRAINING DESIGN

  • SHERPA’s archetypes show improvement concentrated in matched teacher–student pairs and lower complaint rates there, unlike prompted students.The comparison measures selectivity by differentiated improvement and informativeness by differentiated complaint rate.Source: §3.4 · p. 8, Figure 3 · p. 8

4 RELATED WORK

Prior work evaluates, structures, or trains LLM teaching, but often specifies desired teacher behavior through demonstrations, preferences, or explicit criteria. Sherpa instead represents instructional needs through students and trains on their improvement.

Evaluating LLM teachers

  • Teaching benchmarks assess pedagogical skills, while system-level approaches add strategy steering, course planning, reflection, or tutor suggestions; this work trains the underlying model for adaptive guidance.Source: §4 · p. 9

Training LLM teachers

  • Supervised fine-tuning, preference optimization, and online reinforcement learning use demonstrations, comparisons, or explicit scoring rules to specify desirable teaching behavior.Sherpa instead treats teaching requirements as learner preferences and trains on student improvement.Source: §4 · p. 9

Student simulation for teacher training

  • Student simulation can fail to ensure responsiveness to instruction, and unlearning can leave targeted knowledge accessible or affect retained capabilities.Sherpa encodes preferences through adaptive gates in a less capable student model, without extra parameter updates to mimic errors or suppress knowledge.Source: §4 · p. 9

5 CONCLUSION

The conclusion presents SHERPA as a framework for learning adaptive guidance through student archetypes, with gains on student models, an external benchmark, and human judgments. It identifies more specific preferences and gradual inference of student needs as directions for future work.

5 CONCLUSION

  • SHERPA separates teacher and student, using gates to represent learning preferences and student improvement to train adaptive guidance.The authors report improved outcomes across archetypes, gains on an external teaching benchmark, and human preference over the base model.Source: §5 · p. 9
  • Future work could represent combinations or other preferences and have teachers infer preferences that students may not know themselves.The authors frame real-world support for human students as a further step.Source: §5 · pp. 9–10

A OBJECTIVE AND DERIVATION

The standard objective weights turn-level policy-gradient terms with masked returns, then applies centering, normalization, and PPO clipping. The asynchronous implementation adds off-policy correction for older rollout samples while clipping relative to a recent policy.

A.1 STANDARD FORMULATION

  • The masked return gives trajectory-level outcome weight only to turns that pass both gates, setting rejected turns’ return to zero.After centering, normalization, and local penalties, a rejected turn may still have a nonzero final advantage.Source: §A.1 · p. 14
  • Turn-level advantages are broadcast across teacher-generated tokens, with leave-one-out centering and batch normalization preceding the standard PPO clipped surrogate.The return mask controls which turns receive trajectory-level outcome return; subsequent optimization follows the standard pipeline.Source: §A.1 · p. 14

A.2 ASYNCHRONOUS IMPLEMENTATION

  • Asynchronous training uses decoupled PPO because rollout samples can come from older policies.Behavior-policy correction accounts for off-policy samples, while clipping constrains updates relative to the recent proximal policy; when behavior and proximal policies coincide, the correction weight is 1 and the objective reduces to standard PPO.Source: §A.2 · pp. 14–15
  • Dual clipping and importance-weight masking provide additional safeguards only when their respective bounds are exceeded.The implementation uses dual-clipping coefficient c = 3 and importance-weight masking threshold κ = 5.Source: §A.2 · p. 15

B.1 TRAINING AND GENERATION SETTINGS

Training uses MATH problems and multiple student archetypes, while evaluations cover archetype performance, MathTutorBench, human judgments, and simulator variants. The section also reports a preparation ablation and the procedures used to measure outcomes.

B.1 TRAINING AND GENERATION SETTINGS

  • Training uses 759 MATH problems and evaluates on 528; SHERPADP trains on four ID archetypes, while SHERPANP uses the None archetype.Tutoring lasts up to 10 turns, and training rewards improvement only on valid turns passing both gates.Source: §B.1 · p. 15

B.2 EVALUATION ON STUDENT ARCHETYPES

  • Teachers are evaluated on all 528 problems and seven archetypes, with question-level bootstrap intervals quantifying sampling uncertainty.The bootstrap uses 10,000 resamples, retaining all archetypes and outcomes for each sampled problem and pairing comparisons across teachers.Source: §B.2 · pp. 15–16

B.3 MATHTUTORBENCH EVALUATION DETAILS

  • MathTutorBench measures problem solving and mistake correction by accuracy, while four response-generation tasks use Ped-RM win rates.Pedagogy Average is the mean of the four win rates; other measures include BLEU, F1, and micro-F1.Source: §B.3 · p. 16

B.4 HUMAN STUDY DETAILS

  • The human study compares SHERPADP with the untrained Qwen3-8B backbone using blinded, randomized response pairs.Ninety-six high school teachers provided 960 judgments across ten items each, with three judgments per item.Source: §B.4 · p. 16

B.5 STUDENT SIMULATOR ANALYSIS SETTINGS

  • The simulator analysis samples 50 problems and compares archetypes with gate-rejection responses against a prompted variant without external adaptive gating.Rejected messages in the archetype setup receive one of six fixed expressions of lack of understanding.Source: §B.5 · p. 17

B.6 TRAINING-OBJECTIVE COMPARISON SETTINGS

  • The training-objective comparison includes the base model, SHERPADP, and two gate-penalty variants with magnitudes 0.25 and 0.5.Both variants use the SHERPADP training setup and apply penalties for gate violations.Source: §B.6 · p. 17

C.1 EFFECT OF PREPARATION

  • Preparation raises overall student accuracy from 45.3% to 48.6%, a gain of 3.3 percentage points.ID accuracy rises from 44.9% to 49.1%, while OOD accuracy rises from 45.9% to 47.9%.Source: §C.1 · p. 17, Table 4 · p. 17

C.2.1 TRAINING ON INDIVIDUAL ARCHETYPES

Training across ID archetypes yields the strongest overall and OOD student performance among the Qwen3-8B teachers, with MathTutorBench results showing gains in pedagogy measures despite Gemini’s stronger math and diagnosis scores.

C.2.1 TRAINING ON INDIVIDUAL ARCHETYPES

  • 69.0% versus 70.1%: SHERPADP’s overall student test accuracy is comparable to Gemini 3.8 Flash, and each single-archetype teacher exceeds SHERPANP’s 52.0%.Among Qwen3-8B teachers, SHERPADP has the highest OOD and overall averages, 65.1% and 69.0%, while Subgoal has the highest ID average at 74.5%.Source: §C.2.1 · p. 17, Table 5 · p. 18
  • 79.2% versus 72.4%: SHERPADP’s MathTutorBench Pedagogy Average exceeds Gemini’s, and training raises the Qwen3-8B score from 52.5% to 79.2%.SHERPADP outperforms single-archetype teachers on every teacher-response-generation metric, while Gemini scores higher on mathematical problem-solving and diagnosis tasks.Source: §C.2.1 · p. 18, Table 6 · p. 18
  • Training raises Qwen3-8B’s instruction-following win rate from 74.0% to 90.3%, exceeding Gemini’s 69.9%.Gains also extend to the hard setting.Source: §C.2.1 · p. 18

C.2.2 FURTHER ANALYSIS OF TEACHING GAINS

SHERPADP combines preference-satisfying messages with useful instruction and sustained interaction, achieving greater overall improvement with fewer accepted turns than the base model and SHERPANP. It also transfers to Gemma and Llama student backbones, where it achieves the highest accuracy for every archetype.

C.2.2 FURTHER ANALYSIS OF TEACHING GAINS

  • Preference satisfaction counts the fraction of messages passing both guidance and adaptive gates, averaged equally across trajectories.Teaching quality is assessed through one accepted round’s student improvement and the average accepted teaching turns per trajectory.Source: §C.2.2 · pp. 18–19
  • SHERPADP achieves greater overall improvement with fewer accepted turns than the base model and SHERPANP, while combining the highest preference satisfaction with strong single-round teaching.The analysis links teaching gains to preference satisfaction, useful instruction, and sustaining interaction.Source: §C.2.2 · pp. 19–20, Table 10 · p. 19
  • Contrast has the highest single-round improvement but delivers 1.0 effective turns per trajectory versus Attempt’s 1.7, and achieves lower overall improvement.It retains a comparable single-round effect to Attempt while delivering fewer effective turns.Source: §C.2.2 · p. 20

C.2.3 GENERALIZATION TO UNSEEN STUDENT BACKBONES

  • With Gemma, overall accuracy rises from 14.3% to 21.7%, and with Llama from 37.5% to 53.5%; SHERPADP leads for every archetype on both backbones.These results indicate transfer beyond the Qwen3-1.7B student used in training.Source: §C.2.3 · p. 20, Table 11 · p. 20

C.2.4 MULTIPLE TRAINING RUN

Across three training seeds, the standard model has mean overall accuracy of 68.9±2.0%. SHERPA also improves student accuracy across all seven archetypes when a Ministral teacher is trained.

C.2.4 MULTIPLE TRAINING RUN

  • Three independent runs with different random seeds use the same standard-model training setup.The first run is the one reported in Table 1.Source: §C.2.4 · p. 20, Table 12 · p. 20
  • Across the three seeds, mean overall accuracy is 68.9±2.0%, with ID accuracy of 72.2±1.6% and OOD accuracy of 64.4±2.6%.The reported ± values are sample standard deviations across training seeds.Source: §C.2.4 · p. 20, Table 12 · p. 20

C.2.5 TRANSFER ACROSS TEACHER MODEL FAMILIES

  • SHERPA training with a Ministral-3-8B-Instruct teacher raises overall student accuracy from 39.6% to 55.0% across all seven archetypes.ID accuracy rises from 39.5% to 59.4%, and OOD accuracy from 39.8% to 49.2%.Source: §C.2.5 · p. 21, Table 13 · p. 21

C.3 ADDITIONAL STUDENT SIMULATOR ANALYSIS

The three student backbones differ in how much they improve under fixed teaching strategies, but show a similar strategy-response pattern. Prompted students also tend to continue solving after a mismatched instruction rather than return only a complaint.

C.3 ADDITIONAL STUDENT SIMULATOR ANALYSIS

  • Across Qwen3, Gemma, and Llama students, improvement magnitude differs substantially, but the pattern across six fixed strategies is similar.Gemma gains much less than the other two backbones; changing model family does not change which strategies it responds to.Source: §C.3 · p. 21, Figure 5 · p. 21
  • Prompted students include a complaint on mismatched strategy-preference pairs at a similar rate to the gated design, 65.3% versus 76.9%.The two rates are identical by design for the gated archetypes because rejected instructions are replaced by the scripted complaint alone.Source: §C.3 · p. 21, Table 14 · p. 21
  • Only 28.9% of prompted-student replies are complaint-only; the complaint is usually embedded in a continued solution.Thus, prompted students keep working even when an instruction misses their assigned preference.Source: §C.3 · p. 21, Table 14 · p. 21

C.4 ANALYSIS OF DIFFERENT TRAINING DESIGNS

Stronger gate penalties do not improve teaching performance or generalization, and student-improvement training yields substantial gains with or without turn-level masking. Masking concentrates the reward on accepted turns and reduces dialogue length and generated-token use.

C.4 ANALYSIS OF DIFFERENT TRAINING DESIGNS

  • At penalty 0.25, overall accuracy is 69.1% versus 69.0% for SHERPADP; penalty 0.5 lowers overall accuracy to 57.5%.The 0.25 penalty slightly raises ID accuracy and lowers OOD accuracy, while 0.5 also lowers all four MathTutorBench pedagogical scores.Source: §C.4 · p. 22, Table 15 · p. 22
  • Figure 6 shows PedagogicalRL obtains most leakage gain within 200 updates while helpfulness stays similar; SHERPA’s gate pass rate rises more gradually.The authors suggest the more gradual rise may help explain SHERPA’s better generalization than optimization of predefined teaching metrics.Source: Figure 6 · p. 22, §C.4 · p. 22

C.5 EFFECT OF TURN-LEVEL MASKING

  • Masking concentrates student-improvement reward on valid, gate-accepted turns and uses approximately 38% fewer generated training tokens at comparable performance.Over 500 updates, masked training averages 5.0 turns per episode versus 8.1 without masking.Source: §C.5 · p. 23
  • After 500 updates, training without masking achieves 58.6% overall student accuracy, close to 59.5% with masking; both exceed the listed baselines.The baselines are SHERPANP at 51.8%, PedagogicalRL at 51.5%, and the untrained backbone at 48.6%.Source: Table 16 · p. 23, §C.5 · p. 23

D PROMPTS AND TEMPLATES

Appendix D collects prompts and templates for training, evaluation, and human and simulator studies. They specify how teachers and students interact and how teaching responses are checked.

D.1.1 Guidance Gate Prompt

  • The guidance gate rejects a teacher message when a number or arithmetic expression in it equals the ground-truth answer.It extracts numeric content, computes arithmetic expressions, and compares the resulting values with the ground truth.Source: §D.1.1 · p. 24

D.1.2 Adaptive Gate Prompts

  • Six adaptive gates judge whether a tutor follows a stated student preference, using PASS or FAIL rather than judging mathematical correctness.The preferences cover attempt diagnosis, causal justification, subgoal decomposition, step demonstration, independent verification, and contrastive comparison.Source: §D.1.2 · pp. 25–26

D.1.3 Adaptive-Gate Feedback

  • Preference-specific feedback prompts redirect rejected responses; attempt-diagnosis feedback asks for a specific part of the student’s work and why it is right, wrong, or missing.The training and main archetype evaluation use this feedback pool; the simulator analysis uses a bare complaint pool listed in Appendix D.5.Source: §D.1.3 · p. 26

D.2 Training Prompts and Templates

  • The training prompt gives the teacher a 10-turn budget, asks it to prepare a solution draft, then instructs it to infer student needs and teach without revealing the answer.The student later attempts the problem from scratch, and the unaided baseline receives the same testing request without interaction history.Source: §D.2 · pp. 28–29

D.3 Student Archetype Evaluation Prompts; D.3.1 Answer-Correctness Judge

  • Table 1 evaluation uses the training and testing prompts, while an answer judge checks whether the extracted student answer is mathematically equivalent to the reference.Untrained Qwen3-8B and Gemini receive a direct-reply opening prompt, unlike trained teachers’ tagged tutoring format.Source: §D.3 · p. 29, §D.3.1 · p. 29

D.4.1 Student Request Generation

  • Student-request generation prompts Gemini to add a natural-sounding request expressing one of four needs, including subgoal decomposition and error anticipation.The listed needs also include contrastive comparison and causal justification.Source: §D.4.1 · p. 30

D.4.2 Teacher Response Generation; D.4.3 Annotation Instructions

  • For human response comparisons, teachers receive the conversation and student request, while annotators see randomized A/B replies and judge which teaches better.Annotators are instructed not to rely on length, formatting, or other surface features alone.Source: §D.4.2 · p. 30, §D.4.3 · p. 31

D.5.1 Fixed Teacher Strategies; D.5.2 Student Prompts and Feedback

  • Simulator comparisons include fixed teaching strategies and gated or prompted students; a different-backbone comparison assigns no preference and uses no adaptive gate.Fixed strategies are maintained throughout the conversation, and prompted students either respond to the tutor or return a complaint tied to their preference.Source: §D.5.1 · p. 31, §D.5.2 · p. 32
Loading 2610.08778v1…