PaperlayerView exact revision on arXiv

Source-linked AI summary

The learner who does not learn: when optimizing a pedagogical metric degrades LLM tutoring

Daniel Domínguez Figaredo, Rafael Fernández De la Cruz

arXiv:2610.12125v1cs.CY

TL;DR

The paper asks whether optimizing a metric of pedagogical adaptivity improves tutoring, given that evaluations often score discrete moves independently of learner conditions. It tests metric-based correction with a tutor audit, proxy fine-tuning, educator ratings, and weight analysis. The metric’s improvement accompanied worse educator ratings, and the authors conclude that measurement validity does not imply optimization validity.

  • Problem

    Current evaluations often assess tutoring moves independently of learner conditions, leaving it unclear whether tutors adapt instruction to a learner’s situation.

    Source: §1 · p. 2

  • Method

    The study scores tutor decisions against learning-situation conditions, fine-tunes an open-weights proxy on weak cases, obtains blind educator ratings, and analyzes model weights.

    Source: §3.1.2 · p. 7, §3.2 · p. 9

  • Results

    The metric improved while the tutor performed worse; educators identified repetition that the metric could not represent, and weight analysis showed the correction rewrote outputs rather than reasoning.

    Source: §5.4 · pp. 21–22

  • Takeaways & Limitations

    Measurement validity does not imply optimization validity, and benchmarks for AI-tutor instruction should preserve the dynamics of a changing learner.

    Source: §1 · p. 3, §5.4 · p. 22

    The authors propose replication on a second open-weights model and cases based on a real curriculum, with in-service teachers evaluating and sequence-level items built into the instrument.

    Source: §5.4 · p. 21

Abstract

from arXiv · show

It is assumed that a natural way to improve the pedagogical quality of large language model tutors is to define a metric of instructional performance and fine-tune the model against it. To test this strategy, we designed a metric of pedagogical adaptivity that scores each instructional decision in a learning sequence against the conditions of the learning situation, which is the standard used for automated pedagogical scoring. We audited a frontier tutor across 2,000 learner scenarios, corrected its weakest cases by fine-tuning an open-weights proxy, and asked 31 trained educators to rate the pedagogical alignment of the outputs blind, before and after correction. The metric increased from +0.05 to +0.42 for the corrected cases, while the expert ratings decreased from 4.46 to 3.03, with the unmodified controls remaining unchanged and a base-proxy control ruling out the change of model. The tutor performed worse because any metric that scores decisions independently and averages them is maximized by repeating the single best decision, and the fine-tuned model collapsed to that exact optimum in every case, in and out of sample. Educators identified the repetition, which such metrics cannot represent, and preserving the learner's trajectory in the score reduced, but did not reverse, the metric's verdict. Weight analysis traced the correction to the model's output projection, where it had memorized its training strings rather than learned to adapt. We conclude that measurement validity does not imply optimization validity, and we derive design principles for benchmarks that assess or train AI-tutor instruction.

1. Introduction

The paper asks whether optimizing a metric of pedagogical adaptivity improves tutoring, given that current evaluations often score discrete tutoring skills rather than adaptation to learners’ conditions. It audits a tutor, fine-tunes an open-weights proxy on weak cases, and compares automated scores with educators’ judgments and model-weight changes.

Motivation

  • Existing evaluations often assess tutoring competencies independently of learner conditions and leave internal computation unexamined.The study combines metric scoring, educator judgments, and interpretability analysis to examine the intervention and its effects.Source: §1 · p. 2

Research questions

  • The study tests whether fine-tuning against a metric of pedagogical adaptivity improves the instruction that metric measures.It also asks how educator judgments correspond to the metric, what metric property explains any gap, and where the correction changes the model.Source: §1 · pp. 2–3

Approach

  • The Pedagogical Adaptivity Index scores five instructional decision points against conditions of the learning situation.The study audits a frontier tutor across 2,000 scenarios, fine-tunes an open-weights proxy on weak cases, and has trained educators blindly review original and corrected outputs.Source: §1 · pp. 2–3

Contributions

  • The paper argues that metric-based correction can degrade the pedagogical behavior the metric was designed to measure, while automated evaluation alone misses that degradation.It also identifies a structural property that determines optimization safety and derives benchmark design principles.Source: §1 · p. 3

2. Theoretical background

The paper frames pedagogical adaptivity as instruction contingent on learner and task conditions that change during learning. It situates its study amid evaluations and optimization methods that largely treat tutoring moves independently of those conditions.

2.1 Pedagogical adaptivity as contingent instruction

  • Pedagogical adaptivity is the degree to which an instructional agent calibrates decisions to learning conditions, with no single policy optimal across all conditions.The framework grounds decisions in theories of scaffolding, learner schemas, and engagement matched to task and learner.Source: §2.1 · p. 4
  • Competent instruction responds to a learner’s changing trajectory, so a sequence cannot be understood as independent decisions calibrated to an unchanged learner.Scaffolding should fade and transfer responsibility; knowledge state and the learner’s zone of development also change as instruction proceeds.Source: §2.1 · pp. 3–4
  • The framework specifies knowledge state, learning stage, and learning context as learner conditions, each understood as a position on a trajectory rather than a permanent label.It distinguishes novice, informed, and misinformed knowledge states; four stages toward competence; and guided, collaborative, or autonomous contexts.Source: §2.1 · p. 4

2.2 Evaluating and optimizing LLM tutors

  • Prior tutor evaluations and optimization efforts largely assess discrete moves independently, leaving adaptation to learner knowledge, stage, and context underexamined.The paper also notes that existing human–AI comparisons do not establish how expert and automated judgments diverge across pedagogical dimensions and conditions.Source: §2.2 · p. 5

3. Methods

The methods define the Pedagogical Adaptivity Index (PAI), elicit structured tutor sequences, and score decisions against conditions of a learning situation. The study compares corrective fine-tuning and controls, obtains blind educator ratings, and analyzes model weights and condition recovery.

3.1 The Pedagogical Adaptivity Index

  • The PAI scores alignment between a tutor response and a learning situation on a −1.0 to +1.0 scale.The study presents the instrument through its elicitation and scoring procedures.Source: §3.1 · p. 5

3.1.1 Elicitation

  • Each case narrative describes a learner, prior knowledge, objective, and circumstances; the fixed task elicits five activities with five selections and brief descriptions per activity.The five decision points each offer four ordered options, and the PAI scores selections rather than descriptions.Source: §3.1.1 · pp. 5–6, Figure 1 · p. 7

3.1.2 Scoring

  • The PAI averages applicable condition–option matrix scores for each decision, then aggregates them so a response score equals the mean of its 25 decisions or five decision-point scores.Each matrix assigns graded appropriateness scores to an option under a conditioning-variable value.Source: §3.1.2 · p. 7, Table 2 · p. 7, Figure 2 · p. 8

3.1.3 Structural properties of the index

  • Because the PAI scores decisions independently against conditions fixed for the case, it cannot represent whether a sequence varies, progresses, or coheres.Repetition rate and selection entropy were computed separately to register whether a sequence varies.Source: §3.1.3 · p. 8

3.2 Research design

  • The design compares original outputs, greedy and diversity-preserving corrective fine-tunes, and an un-fine-tuned proxy, with case type as a blocking factor.The base-proxy arm holds generator, cases, prompt, and decoding fixed; controls remain unchanged, though no educator rated the base proxy.Source: §3.2 · p. 9

3.3 Models and case corpus

  • The audit uses Gemini 3.1 Pro Preview and Qwen3-32B as proxy, with 2,000 balanced scenarios and separate intervention, control, training, and out-of-sample sets.The mechanistic claims concern fine-tuning an open-weights instruction-tuned transformer, not a commercial tutor.Source: §3.3 · p. 9

3.4 Corrective fine-tuning; 3.5 Expert evaluation

  • Greedy supervision selects metric-maximizing options at every decision, while diversity-preserving supervision corrects one decision point per example and retains the other original decisions.Both use LoRA on the proxy; the greedy adapter rank is swept over r ∈ {1, 4, 8, 16}. Educators rated the original outputs and then greedy-corrected intervention outputs while controls stayed unchanged.Source: §3.4 · pp. 9–10, §3.5 · p. 10

3.6 Representational analyses; 3.7 Condition-recovery probe; 3.8 Data analysis

  • Weight analyses test update localization and overlap between correction subspaces, while a condition-recovery probe compares model classifications with corpus labels and text-based predictions.The analyses also specify rank, spectral profile, effective rank, principal angles, and greedy-update energy; statistical tests compare metric, sequence measures, ratings, and coded justifications.Source: §3.6 · pp. 10–11, §3.7 · p. 11, §3.8 · p. 11

4. Results

Fine-tuning raised PAI while lowering expert ratings, as the metric’s independently scored decisions rewarded repetitive choices. The results locate this collapse in output-stage memorization and show that unrecovered learner conditions predict vulnerability to it.

4.1 Baseline audit

  • The audited tutor’s mean PAI was +0.29 across 2,000 scenarios; support structure and cognitive task design were weakest above their random baselines.The respective gaps above random were +0.04 for support structure and +0.06 for cognitive task design.Source: §4.1 · p. 12, Figure 3 · p. 12

4.2 Expert judgment against the metric, before and after correction

  • +0.37 PAI gain (from +0.05 to +0.42) on 30 intervention cases accompanied a fall in expert ratings from 4.46 to 3.03; unmodified controls remained unchanged.At baseline, expert ratings did not track PAI (ρ = −0.15, p = .39).Source: Figure 4 · p. 13, §4.2 · p. 13

4.3 The degenerate optimum

  • Diversity-preserving adapters recovered at most about a third of greedy’s gain, while trajectory-aware rescoring reduced the intervention advantage to +0.07 without significantly reversing it.The rescored comparison was 0.34 versus 0.27 (p = .047); across faster and slower schedules, the advantage ranged from −0.02 to +0.15.Source: Figure 4 · p. 13, §4.3 · p. 16

4.2 Expert judgment against the metric, before and after correction

  • Human coders identified monotony in 19–25% of weakest-aspect critiques for corrected intervention outputs, and those outputs were rated 2.2 versus 3.2 without a monotony critique.The repetition concerned instructional choices rather than the differently worded descriptions.Source: §4.2 · pp. 13–14, Figure 5 · p. 14

4.3 The degenerate optimum

  • The independently scored metric made repetition the optimum, and greedy fine-tuning produced fully monotone sequences in 100% of tested cases, including 200 unseen cases.The unseen-case greedy model had mean PAI 0.42, versus 0.30 for the base proxy.Source: §4.3 · pp. 14–15, Figure 6 · p. 15
  • On unseen learners, every corrected model used the same DP2 option, while DP2 scores fell from 0.24 to 0.04 even as overall PAI rose.The base proxy used four distinct DP2 options; the corrected model continued varying its choices on the other four decisions.Source: Figure 6 · p. 15, §4.3 · p. 15

4.4 Locating the correction in the weights

  • At ranks r = 1, 4, 8, and 16, 75–93% of greedy update energy went to the output projection, where corrected outputs reproduced 748 of 750 training-bank strings.Diversity-preserving updates used the output projection less, and shared only 3.3% of the greedy update’s input-side directions.Source: §4.4 · p. 16, Figure 7 · p. 17

4.5 Which conditions are recoverable

  • Learning stage was not recovered (κ = 0.16 for the audited model; 0.05 for the proxy), and condition recoverability predicted collapse vulnerability (ρ = +0.95), not baseline adaptivity.The models’ decisions were scored against a learning-stage condition they did not have; recoverability correlated only ρ = −0.10 with baseline adaptivity.Source: §4.5 · pp. 17–18, Table 4 · p. 18

5. Discussion

The correction raised the pedagogical metric while educators judged the instruction worse, exposing a gap between measurement validity and optimization validity. The discussion traces that gap to independently scored decisions, omitted learner trajectories, and a correction concentrated in the model’s readout, then draws implications and limitations for benchmarks.

5. Discussion

  • The metric rose while educators rated corrected instruction lower, and the model collapsed to repeated decisions both on training cases and unseen cases.Educators identified repetition that the metric could not represent; the correction rewrote output rather than reasoning.Source: §5 · p. 18, §5.4 · pp. 21–22

5.1 Measurement validity does not imply optimization validity

  • A metric can pass construct-validity checks yet be unsafe to optimize because validation observes existing behavior while optimization drives behavior toward an unvalidated argmax.An objective that independently scores activities and averages them is maximized by repeating the single best-scoring activity, regardless of judgment calibration.Source: §5.1 · pp. 18–19
  • The PAI retained situational contingencies but omitted learner dynamics, so its fixed-condition sequence treated repeating one activity as optimal.Allowing learner conditions to change removed repetition as the optimum, but the rescored metric still favored collapsed outputs and did not match educator judgments.Source: §5.1 · p. 19

5.2 The correction only affected the readout

  • Context-independent corrective targets concentrated 75–93% of update energy in the readout, while targets that could not be met that way shifted 66–80% into computation, or 41% for the most variable arm.The depth of representational change tracked objective demands; the authors note they cannot determine whether base-model adaptivity uses learner-condition representations or shallower associations.Source: §5.2 · pp. 19–20

5.3 Implications for benchmarks of AI-tutor instruction

  • The authors propose optimization validity and benchmark rules including verifying recoverability, separating recognition from decision, measuring adaptation as a difference, and publishing the argmax.They present these as additions to existing construct-validity guidance.Source: §5.3 · p. 20, Table 5 · p. 20

5.4 Limitations and conclusions

  • Human-evaluation evidence is qualified because educators did not rate the base proxy, corrected outputs also used generic descriptions, and ratings were in-sample.The design cannot fully apportion the rating decline between collapsed selections and generic descriptions, while behavioral collapse generalized out of sample.Source: §5.4 · p. 21
  • The evaluation used synthetic cases, graduate-student raters, and rule-based coding validated by two coders; one condition was weakly encoded.The authors propose a real-curriculum replication with in-service teachers and sequence-level items built into the instrument.Source: §5.4 · p. 21
  • The study did not establish that removing the orthogonal component preserves metric gains without collapse, and its diversity-preserving training targets cannot be independently verified.Single-run training without a fixed seed also leaves rank comparisons vulnerable to initialization noise; results depend on grouping Bloom’s six levels into three bands.Source: §5.4 · p. 21

Appendix A. The PAI alignment matrices

The appendix presents alignment matrices that score instructional options under conditions relevant to each of five decision points. Scores are derived from applicable tables, whose matrices were specified before scoring and allow multiple options to receive positive ratings.

Appendix A. The PAI alignment matrices

B.1 Models

The appendix identifies the models and describes how cases and corrective supervision were assembled. The corpus spans balanced learning scenarios, while corrective targets include both metric-optimal repetition and decision-specific corrections that retain variation.

B.1 Models

  • The study used grok-3-fast to generate case narratives, Gemini 3.1 Pro Preview as the closed-weight audited tutor, and Qwen3-32B as the open-weight proxy.The proxy was fine-tuned with LoRA on Tinker and served for evaluation through vLLM.Source: §B.1 · p. 29

B.2 Case corpus

  • The case corpus sampled 2,000 balanced scenarios across cognitive level, subject family, knowledge state, learning stage, and learning context.Each was rendered as an 80–120-word narrative with condition values implicit, then checked for condition faithfulness and lexical cues to expected answers.Source: §B.2 · p. 29
  • The corrective set contained 589 cases stratified across the ten condition cells where the audited model was weakest.The 30 lowest-PAI intervention cases were contained within that set; eight highest-PAI control cases and 200 disjoint out-of-sample cases were also specified.Source: §B.2 · p. 29

B.3 Corrective supervision

  • Greedy supervision selected the PAI-maximizing option at every decision point and paired it with a learner-independent description, producing repetition 1.000 and entropy 0.000.Source: §B.3 · p. 30
  • Diversity-preserving supervision corrected one decision point at a time while retaining cross-activity variation, with repetition approximately 0.60 and entropy approximately 1.20.Four corrective sets were constructed, one for each decision point.Source: §B.3 · p. 30

B.4 Fine-tuning configuration

The appendix specifies Qwen3-32B fine-tuning with LoRA and reports rank settings, optimizer configuration, and adapter scope. The rank sweep varies effective update scale along with rank because α is fixed.

B.4 Fine-tuning configuration

  • Adapters targeted attention, MLP, and unembedding linear modules, yielding 449 adapted matrices per adapter.Released adapter weights cover all ranks and decision-point variants.Source: §B.4 · pp. 30–31
  • LoRA fine-tuning used Qwen/Qwen3-32B, Adam, batch size 1, and cross-entropy over the final assistant message.The optimizer settings include β₁ = 0.9, β₂ = 0.95, ε = 1 × 10⁻¹², weight decay 0.0, and no gradient clipping.Source: §B.4 · p. 30
  • The greedy sweep used LoRA ranks 1, 4, 8, and 16, while decision-point adapters used rank 8.The training seed was not fixed.Source: §B.4 · p. 30
  • Decision-point adapters used rank 8 for content complexity, cognitive task, support structure, and engagement mode, with sample sizes of 150, 300, 225, and 52, respectively.Their reported final losses were 0.0124, 0.0120, 0.0115, and 0.0168, respectively.Source: §B.4 · p. 30
  • With α fixed at 32, the sweep’s effective scaling factor α/r ranges from 32 at rank 1 to 2 at rank 16, so rank and update scale covary.Accordingly, the rank sweep does not isolate adapter capacity alone.Source: §B.4 · p. 30

B.5 Decoding parameters

The corrected model repeated a single option across activities under different decoding regimes, including out of sample. The appendix reports that generation settings vary by experiment.

B.5 Decoding parameters

  • Generation settings differed by experiment, and all model queries used constrained JSON output for parsing option selections.The appendix lists experiment arms covering the audited tutor, corrective cases, out-of-sample cases, and rated cases.Source: §B.5 · p. 31
  • 100% of corrective cases were fully monotone at every rank, and the same 100% collapse recurred on 200 out-of-sample cases.Under identical sampled settings, the base proxy retained full option diversity with mean selection entropy 1.540.Source: §B.5 · p. 31
  • The repetition appeared under both sampled and greedy evaluation, so it was not attributed to a deterministic decoder or a particular sampling temperature.The corrected model repeated one option across all five activities even when sampling could vary them.Source: §B.5 · p. 31

B.6 Reproducibility

Reported analyses use released materials, and the out-of-sample and rated-case evaluations use deterministic greedy decoding.

B.6 Reproducibility

  • All reported analyses were computed from released adapters, datasets, and scoring code rather than by rerunning training.The released materials reproduce every reported analysis.Source: §B.6 · p. 31
  • Out-of-sample and rated-case evaluation used greedy decoding and was deterministic.The absence of a decoding seed therefore does not affect reproducibility of those results.Source: §B.6 · p. 31
  • The appendix distinguishes reproducibility of reported analyses from rerunning training: analyses rely on released artifacts, not a new training run.The released artifacts include adapters, datasets, and scoring code.Source: §B.6 · p. 31

B.7 Known limitation: training seed

Training used a single run per condition with an unfixed seed, limiting estimates of run-to-run variability. The reported collapse nevertheless recurred at every rank and on disjoint cases.

B.7 Known limitation: training seed

  • 100% of cases collapsed at every rank, including rank one, and the collapse reproduced on 200 cases disjoint from training.The authors say this makes a fortunate single run an implausible explanation.Source: §B.7 · p. 31
  • Each condition was trained once with a random LoRA initialization because the training seed was not fixed.The configuration therefore does not reproduce training bit for bit, although released adapter weights reproduce the reported analyses exactly.Source: §B.7 · p. 31
  • With one run per condition, the study cannot estimate run-to-run variance or formally distinguish small rank differences from initialization noise.The principal collapse finding does not depend on such rank differences.Source: §B.7 · p. 31

B.8 Note on training loss

The corrective targets were nearly memorized, consistent with an output-projection readout edit. The appendices describe the blinded educator-rating instrument and a codebook for identifying what educators discussed within and beyond the PAI framework.

B.8 Note on training loss

  • Approximately 0.003 final training loss at every rank indicates near-memorization of 589 monotone corrective targets, consistent with an output-projection readout edit.The passage describes the near-perfect fit and sequence collapse as two descriptions of the same outcome.Source: §B.8 · p. 32

Appendix C. The expert-rating instrument

  • The instrument also included two free-text prompts asking raters to explain a plan’s weakest and strongest aspects for the particular learner.Source: Appendix C · p. 32
  • The six 1–5 items assessed overall alignment, difficulty and depth, learning tasks, guidance and support, engagement, and subject-matter approach.The PAI mapping was used only in analysis and was not shown to raters.Source: Appendix C · p. 32
  • Raters saw learner narratives and five-activity plans, but not PAI scores, alignment matrices, or intervention status; 31 educators completed both rounds.Each case received four to eleven independent ratings, averaging 6.5.Source: Appendix C · p. 32

Appendix D. Qualitative codebook

  • The qualitative codebook separated educator comments mapped to the five PAI decision points from comments about unmeasured sequence properties such as monotony, pacing, coherence, personalization, and affective fit.The latter properties were not scored by the PAI.Source: Appendix D · p. 33
  • Two educators independently coded a stratified 20% subsample of weakest-aspect texts (n = 99), blind to automated codes; only supported codes entered the analysis.The codebook, coding script, and coding sheets were released with the code.Source: Appendix D · p. 33
Loading 2610.12125v1…