PaperlayerView exact revision on arXiv

Source-linked AI summary

StudentBench: AI and human tutoring yield equivalent GRE learning gains

Curtis Northcutt, Inaara Hasmani, Kevin Feng, Trevor Khangi, Andreas Plesner, Jonas Mueller

arXiv:2609.28470v1cs.AIcs.CY

TL;DR

Evidence that AI can augment human capabilities has often emphasized model performance, leaving limited measurement of real students’ learning gains after tutoring. StudentBench evaluates LLM tutoring and teaching materials, finding equivalent GRE learning gains for pooled AI and human tutors and a tested AI tutor achieving that equivalence at much lower cost.

  • Problem

    Few LLM benchmarks measure whether real students learn after tutoring, leaving limited evidence about AI’s ability to augment human capabilities.

    Source: §8.2 · p. 10

  • Method

    StudentBench measures GRE learning gains after one-hour AI or human tutoring and uses expert evaluations of AI-generated teaching materials.

    Source: §1 · pp. 1–2

  • Results

    AI and human tutoring produced equivalent combined GRE learning gains (p=.015), and Gemma 4 31B achieved equivalent gains at 918× lower cost per percentage point than human tutoring.

    Source: §3 · p. 5, §10 · p. 10

  • Takeaways & Limitations

    StudentBench provides evaluation paradigms for measuring real student learning gains and distinguishing AI tutors across teaching, cost, and engagement dimensions.

    Source: §10 · p. 10

    The study measured immediate learning gains but did not evaluate whether they persist over months.

    Source: §9 · p. 10

Abstract

from arXiv · show

Artificial intelligence offers an unprecedented opportunity to augment human capabilities, yet progress at the frontier has focused primarily on advancing model capabilities. We introduce StudentBench, a suite of AI teaching evaluations and a public platform that enables large-scale data collection with over 175,000 student-AI messages to study whether large language models (LLMs) produce learning gains equivalent to human tutoring. Using StudentBench, we measured learning gains on Quantitative and Verbal GRE questions across 2,383 human participants receiving AI tutoring, human tutoring, or no tutoring. We establish that AI tutoring is statistically equivalent to expert human tutoring for GRE learning gains (p = .015), and in five of the seven GRE domains, the best performing AI tutor surpassed the human tutor, on average. In a second study, expert human tutors compared LLM-generated lesson plans and practice problems through 2,028 pairwise rubric evaluations. Together, the two studies clearly separate AI tutors across: (1) lesson planning, (2) practice-problem creation, (3) conversational pedagogy, (4) cost, and (5) engagement. Surprisingly, one AI tutor achieved learning gains equivalent to human tutoring (p = .044) at 918 times lower cost (USD 0.0052 for AI versus USD 4.81 for human, per percentage point gained). For Quantitative GRE sessions, faster AI replies correlated with more student messages, more messages with more correct practice, and more correct practice with larger learning gains (all p < .002). The StudentBench platform is freely available at https://studentbench.org.

1 The pursuit of human-level tutoring

The paper asks whether teaching technology can support recursive human self-improvement by testing whether LLM tutors produce learning gains comparable to expert human tutors. StudentBench evaluates tutoring on GRE questions and reports comparable gains, alongside substantial differences across tutors and teaching dimensions.

1 The pursuit of human-level tutoring

  • Across 2,383 participants, pooled AI and human tutors produced statistically equivalent GRE learning gains (p=.015) after one-hour tutoring sessions.The study tests the first condition for recursive human self-improvement: whether technology improves human capabilities.Source: §1 · p. 2
  • In five of seven GRE domains, the best-performing AI tutor surpassed the human tutor on average, while human tutoring retained the highest whole-section mean in Verbal.Several AI tutors had higher mean gains than human tutoring in Quantitative tests.Source: Figure 1 · p. 2, §1 · p. 3
  • One AI tutor achieved gains statistically equivalent to expert human tutoring (p=.044) at approximately 918× lower cost per percentage point of learning gain.Its mean inference cost was $0.067 for a full tutoring session.Source: §1 · p. 3
  • StudentBench evaluates lesson planning, practice-problem creation, and conversational pedagogy, and uses expert reviews of teaching materials and recorded tutoring conversations to create leaderboards.The second study compared LLM-generated lesson plans and practice problems through pairwise rubric evaluations.Source: §1 · p. 3
  • For Quantitative GRE sessions, faster AI replies were associated with more student engagement, more correct practice, and larger learning gains.The paper presents these relationships as correlations.Source: §1 · p. 3
  • StudentBench introduces evaluations of real student learning gains and makes its platform, study data, and reproducibility code available.The platform is freely available to students.Source: §1 · p. 3

2 Measuring learning and teaching

The study compares learning after AI tutoring, human tutoring, and a no-tutoring control using GRE pre- and post-tests. It also measures the cost of learning and expert evaluations of AI-generated teaching materials.

2 Measuring learning and teaching

  • Students completed a pre-test, spent one hour in an assigned AI, human, or control condition, and then took a post-test with different questions.The control group watched educational videos unrelated to the GRE; tutors received students’ pre-test mistakes but not post-test questions.Source: §2 · pp. 3–4
  • The analysis included 2,469 sessions from 2,383 students after eligibility and data-quality filters were applied.Participants were recruited through Handshake, and AI tutors were randomly assigned with balanced counts as available.Source: §2 · p. 4
  • Former GRE exam creators designed new pre- and post-test questions to match prior GRE exams’ difficulty, coverage, order, categories, and question types.Old GRE questions were avoided because LLMs may have trained on publicly available former questions.Source: §2 · p. 4
  • The AI tutors used minimal software scaffolding and low-guidance prompts, so the results characterize performance under that setup; further prompt adjustment and scaffolding may improve it.Each tutor used prompts for lesson planning/problem creation and for interactive tutoring.Source: §2 · p. 4
  • Learning gain is post-test score minus pre-test score in percentage points, with ANCOVA adjustment for pre-test differences and specified equivalence tests for AI–human comparisons.Equivalence requires the entire 90% confidence interval to fall within the stated bounds; the combined-GRE ±0.25-SD margin corresponds to ±4.09 percentage points.Source: §2 · p. 4
  • In a second study, 51 expert tutors completed 2,028 pairwise reviews of AI-generated lesson plans and embedded practice problems based on 381 student pre-tests.Reviewers answered eight comparison questions per pair: five about lesson planning and three about practice problems.Source: §2 · p. 4

3 AI and human tutoring yield significant and comparable score gains

Across combined Quantitative and Verbal GRE sessions, pooled AI tutoring and human tutoring produced statistically equivalent learning gains. AI tutoring also outperformed the no-tutoring control, while results varied by section and domain.

3 AI and human tutoring yield significant and comparable score gains

  • AI and human tutoring had equivalent combined-section mean learning gains (p=.015; ±0.25-SD bounds), with an adjusted AI−human difference of −0.58 percentage points (90% CI [−2.18, 1.03]).Equivalence also held under the tighter ±0.20-SD bounds (p=.023); it was established for Quantitative but not Verbal at ±0.25 SD.Source: §3 · p. 5
  • 6.86 percentage points in Quantitative and 5.47 in Verbal were adjusted AI−control learning-gain differences; across both sections, the difference was 6.15 percentage points.The combined difference corresponds to about 1.5–2 more correct answers out of 27 than in the control condition.Source: §3 · p. 5
  • The best AI tutors had higher mean gains than humans in three of four Quantitative domains, while human mean gains exceeded pooled AI gains in all three Verbal domains.The figure traces the highest AI mean learning gain in each domain, with a different tutor leading each domain.Source: §3 · p. 5
  • Among the top 25% of Quantitative pre-test performers, four of the five highest mean gains came from Gemini tutors, led by Gemini 3.5 Flash.This exploratory analysis used item response theory to estimate pre-test proficiency.Source: §3 · p. 5

4 Three evaluations of AI teaching

The section evaluates AI teaching through expert ratings of lesson plans and practice problems and transcript-based measures of conversational pedagogy. It finds differences among AI models across these evaluations.

4 Three evaluations of AI teaching

  • Experts rated lesson planning and practice-problem creation separately, while a third evaluation measured pedagogical characteristics in tutoring conversations.The two expert-rating leaderboards use five planning questions and three practice-problem questions; the conversation evaluation uses six fixed indicators.Source: §4 · p. 5, §4.1 · p. 5

4.1 Lesson planning and practice-problem creation

  • Anthropic models, particularly Opus, were preferred for lesson planning and practice-problem creation and more often displayed the evaluated teaching behaviors.Confidence intervals did not overlap for most model pairs in the lesson-planning and practice-design leaderboards.Source: §4.1 · p. 5
  • Higher expert ratings for practice-problem design were associated with fewer student disputes of AI answers in Quantitative and Combined results.The reported Spearman correlations were ρ=0.71 and 0.77, respectively.Source: §4.1 · p. 5

4.2 Conversational pedagogy

  • AI tutors clustered by model family on the conversation indicators, with distinct vertical bands suggesting shared provider-level training practices.The indicator score averages six binary transcript rules; the authors note a bias toward sessions with more turns or more verbose AI tutors.Source: §4.2 · p. 6

5 AI tutoring cost, latency, and engagement

The section compares learning gains with AI tutoring cost and reply time, and examines how reply time relates to student dialogue. It reports substantial variation across AI tutors and a latency–engagement association.

5 AI tutoring cost, latency, and engagement

  • The section examines the Pareto frontier for learning gain, cost, and latency, using session data and a $75-per-hour human tutoring reference.AI tutoring cost includes recorded or reconstructed inference costs across lesson planning, interactive chat tutoring, and practice-problem generation.Source: §5 · p. 6, §5.1 · p. 6, Figure 4 · p. 7

5.1 Cost of learning

  • Gemini 3.5 Flash had a learning gain similar to GPT-5.5 Pro while costing 20× less; all AI tutors on the Pareto frontier averaged under $5 per session.Across 12 AI tutors, mean inference cost ranged from $0.067 per session for Gemma 4 31B to $21.24 for GPT-5.5 Pro.Source: §5.1 · p. 6
  • Gemini 3.1 Pro had higher mean learning gain, lower inference cost, and faster replies than GPT-5.5 Pro.This is a reported example of one AI tutor strictly Pareto-dominating another.Source: §5.1 · p. 7

5.2 Faster replies are associated with more student dialogue

  • Across 12 AI tutors, lower reply latency was strongly associated with higher student engagement (Spearman ρ=−0.81, p=.0056).Reply times ranged from 1.9 seconds for GPT-5.4 mini to 31.0 seconds for GPT-5.5 Pro.Source: §5.2 · p. 7

6 The cost per percentage point of learning gain

This section introduces cost per percentage point of learning gain to compare AI and human tutoring. One AI tutor achieved equivalent gains at substantially lower cost.

6 The cost per percentage point of learning gain

  • Cost per percentage point of learning gain is mean session cost divided by mean learning gain, providing a common basis for comparing AI and human tutoring costs.The paper measures immediate GRE learning gains.Source: §6 · p. 8
  • The study tested whether each AI tutor’s learning gains were equivalent to human tutoring; Gemma 4 31B achieved GRE learning gains equivalent to expert human tutoring (p=.044) at 918× lower cost per percentage point of learning gain.Source: §6 · p. 8
  • The study reports cost per percentage point in USD, comparing AI inference cost with a human-tutoring reference.Gemma 4 31B is the 1× cost reference; human tutoring uses a $75-per-hour reference.Source: §6 · p. 8

7 How latency, engagement, and practice relate to learning gain

The section reports a chain of associations among AI reply speed, student activity, correct practice, and learning gains in Quantitative GRE sessions. It notes that this pattern was not statistically significant in Verbal.

7 How latency, engagement, and practice relate to learning gain

  • Exploratory analyses linked learning gains with more correct practice, which was associated with greater student engagement, but had not linked learning gains to an AI-tutor characteristic.The reply-time association with engagement supplied that connection for the subsequent analysis.Source: §7 · p. 8
  • In 1,137 Quantitative AI-tutoring sessions, faster replies were associated with more student engagement, more engagement with more correct practice, and more correct practice with greater learning gains (all p<.002).Regressions account for starting score, AI tutor, and assessment form; the associations were not statistically significant in Verbal.Source: §7 · p. 8
  • Verbal students sent fewer, longer messages despite similar word totals, suggesting message count may capture engagement differently across sections.Figure 6 presents associations in three panels, adjusting for pre-test score, AI tutor, and assessment form.Source: §7 · p. 8, Figure 6 · p. 9
  • StudentBench treats latency as a primary benchmark dimension, with engagement and practice proposed as possible links to learning gains.The authors suggest this may help explain why models weaker on teaching-capability leaderboards can still achieve human-level GRE learning gains.Source: §7 · p. 8

8 Background and related work

Prior teaching technologies and tutoring systems provide context for evaluating whether general-purpose LLMs can support human learning. StudentBench contrasts autonomous AI tutoring with earlier benchmarks by measuring real students’ learning and assessing teaching across multiple dimensions.

8.1 Teaching technologies and human capabilities

  • Teaching technologies have a long history, while computer-based tutoring developed systems that model student knowledge, personalize instruction, and use adaptive dialogue.Examples include cognitive tutors and knowledge tracing, followed by AutoTutor’s adaptive questions, hints, and feedback.Source: §8.1 · p. 9
  • Earlier intelligent tutoring systems sometimes achieved learning outcomes comparable to human tutoring, while StudentBench tests whether minimally scaffolded general-purpose LLMs can also produce substantial gains.Prior reviews reported comparable outcomes.Source: §8.1 · p. 9
  • StudentBench examines tutoring dialogue—including scaffolding, reasoning checks, and interactive questioning—and evaluates lesson planning and practice-problem quality.The evaluated dimensions include concept selection, pacing, alignment with pre-test errors, difficulty calibration, and answer accuracy.Source: §8.1 · p. 9
  • StudentBench relates tutoring dialogue, engagement, and practice activity to subsequent learning gains, since activity alone need not indicate learning or retention.Prior work notes that students may advance through hints without intended reasoning, and more completed problems do not necessarily mean greater retention.Source: §8.1 · pp. 9–10

8.2 AI tutoring and benchmarks

  • StudentBench evaluates autonomous AI tutors against individual human tutoring, extending comparisons of hints, structured tutors, and AI assistance or supervision of human tutors.It compares live tutoring conversations and measures learning with unaided post-tests.Source: §8.2 · p. 10
  • StudentBench measures real students’ learning after live tutoring, unlike benchmarks based on simulated conversations or code submissions, and complements expert ratings of teaching materials.Its learning-gain measure is an unaided post-test.Source: §8.2 · p. 10

9 Limitations of this study

The study measures immediate GRE learning gains within a specific participant and testing setup. Its design does not establish longer-term retention or the added benefit of AI tutoring over independent practice.

9 Limitations of this study

  • Learning gains were measured immediately; persistence over months was not evaluated, and a large longitudinal sample would raise recruitment and follow-up costs.The authors call for longitudinal studies of retention after AI tutoring.Source: §9 · p. 10
  • Participants were English-reading adults recruited through Handshake for a paid study, and deployment across languages, devices, or educational settings was not tested.The leaderboard evaluations characterize expert ratings and observed conversational behaviors, not how much learning occurs.Source: §9 · p. 10
  • Without a control group practicing GRE problems independently, the study cannot separately measure AI tutoring’s added benefit over solo practice.Source: §9 · p. 10

10 Conclusion

StudentBench reports GRE learning gains equivalent to expert human tutoring and introduces separate benchmark leaderboards for AI teaching. The authors connect these findings to human capability augmentation and suggest possible relevance beyond the GRE.

10 Conclusion

  • 918× lower cost per percentage point than human tutoring accompanied equivalent GRE learning gains for Gemma 4 31B among six AI tutors passing individual equivalence tests.The reported AI–human equivalence was p=.015.Source: §10 · p. 10
  • The authors suggest AI tutoring may help with other standardized exams, including the SAT and MCAT, and support the teaching condition proposed for recursive human self-improvement.Source: §10 · p. 10
  • StudentBench introduces five leaderboards separating AI tutors across lesson planning, practice-problem creation, conversational pedagogy, cost, and engagement.Source: §10 · p. 10
  • The study contributes infrastructure and data collection methods for evaluating how AI can uplift, rather than replace, human intelligence.Source: §10 · p. 10

Ethics statement

The ethics statement describes consent, privacy protections, funding, and the study’s research-only status. The population and study conditions describe participant recruitment, GRE assessment construction, and procedural choices.

Ethics statement

  • Participants consented to research use under BRANY SBER IRB approval; public data omit identity mappings, individual locations, and audio/video recordings.Handshake AI funded the work, which did not alter official exam records or admissions decisions.Source: Ethics statement · p. 11

A.1 Population and study conditions

  • The analysis included 2,383 adults aged 18–67 recruited through Handshake, with participants offered $50 regardless of test performance.Invitations went to over 70,000 randomly selected students across different majors.Source: §A.1 · p. 16
  • GRE P and Q assessments were counterbalanced between pre-test and post-test, and exam creators matched questions for difficulty, format, and subtopic.New questions were created to match prior GRE exams while avoiding unchanged reuse of original questions.Source: §A.1 · p. 16
  • Each Quantitative test allowed 47 minutes and each Verbal test 41 minutes, alongside a required one-hour assigned condition.Source: §A.1 · p. 16
  • AI tutor assignment used least-fill allocation, and main-study AI sessions shared the same lesson-planning and tutoring prompts within each GRE section.Earlier control-condition data were excluded from the main study.Source: §A.1 · p. 16

A.2 Data quality

This appendix documents the study’s data-quality screens, tutor and interface setup, and standardized GRE tutoring prompts. The prompts guide focused instruction for transferable understanding and practice while avoiding disclosure or speculation about the unseen post-test.

A.2 Data quality

  • Learning comparisons excluded ineligible or non-adherent records using assessment-effort, tutoring-participation, and rapid-submission screens, without filtering on learning outcomes.Screens included completion and pre-test eligibility rules, minimum assessment effort, condition-specific participation requirements, and a rapid-answer threshold applied to either assessment.Source: §A.2 · pp. 16–17

A.3 Study interfaces

  • The interfaces implemented assessment, AI and human tutoring, and expert lesson-plan comparison workflows.Figures A.1–A.5 show the student assessment and tutoring flows and the expert plan-comparison interface.Source: §A.3 · p. 18

A.4 Deployed AI tutors

  • Each GRE section tested 13 AI tutors based on 12 models; tutor versions and reasoning settings varied, and combined leaderboards excluded Sonnet while expert review excluded Gemini 3.6 Flash.The appendix also notes that newer models were not evaluated.Source: §A.4 · pp. 18–19

A.5 Study prompts and supplied context

  • Main-study AI tutors used shared section-specific prompts, while expert comparisons matched prompts within each pair; tutors chose lesson concepts, sequencing, timing, and practice problems.Planning requests included the student’s full pre-test record and expert difficulty ratings, while skill and subskill labels were withheld.Source: §A.5 · pp. 19, 22

A.5.1 Lesson-planning prompt: Quantitative

  • Quantitative lesson planning used raw pre-test responses to infer concepts and difficulty, then required a prioritized, timed curriculum with practice problems and worked solutions aimed at transferable learning.The prompt directed tutors to allocate the full session across concepts according to their importance for the student’s score.Source: §A.5.1 · p. 23

A.5.2 Lesson-planning prompt: Verbal

  • Verbal lesson planning likewise inferred concepts from raw pre-test data and required prioritized, timed instruction with practice problems and worked solutions focused on transferable learning.The prompt covered Text Completion, Sentence Equivalence, and Reading Comprehension, including single-answer and multi-select formats.Source: §A.5.2 · pp. 24–25

A.5.3 Interactive-tutoring prompt: Quantitative

  • The Quantitative interactive-tutoring prompt framed the live chat as direct, one-on-one teaching of transferable concepts and methods rather than delivery of answers.It instructed the tutor to address the learner directly and not claim knowledge of the post-test.Source: §A.5.3 · p. 25
  • Quantitative tutoring is framed around teaching a challenging GRE concept through practice problems and/or a mini-lesson, with genuine understanding and transferable methods as the learning goal.The prompt covers Arithmetic, Algebra, Geometry, and Data Analysis, with GRE formats including Quantitative Comparison, multiple choice, and Numeric Entry.Source: §A.5.3 · p. 26

A.5.4 Interactive-tutoring prompt: Verbal

  • Verbal tutoring is framed around teaching a concept students can apply independently, using practice problems and/or a mini-lesson rather than supplying answers to specific items.The prompt covers Text Completion, Sentence Equivalence, and Reading Comprehension, emphasizing precise reading, vocabulary in context, logical structure, passage evidence, and disciplined elimination.Source: §A.5.4 · p. 26

B Statistical analysis

The appendix details the inference and sensitivity analyses behind the learning-gain comparisons, then extends the results to tutor-level cost, leaderboard separation, and GRE-domain rankings.

B Statistical analysis

  • Holm correction is applied separately to predefined families of learning, preference, and dialogue comparisons; confidence intervals are pointwise unless stated otherwise.Equivalence tests assess whether differences lie within stated bounds, while difference and association tests assess whether contrasts or associations differ from zero.Source: B Statistical analysis · p. 27

B.1 Learning outcomes and contrasts

  • Learning gain is post-test minus pre-test; adjusted condition gains use predicted post-test scores at each session’s observed baseline, with no tutoring as the reference.The combined AI-tutor omnibus test did not detect differences among the 12 shared tutors (p=.755) and does not compare AI with human tutoring or control.Source: §B.1 · pp. 27–28

B.2 Pooled AI–human equivalence

  • −0.58 percentage points was the combined AI −human difference (90% CI [−2.18, 1.03]; TOST p=.015), meeting the ±0.25-SD equivalence bounds.The pooled result also met stricter ±0.20-SD bounds (p=.023); its effective degrees of freedom were 1.59 because human sessions clustered around shared tutors and repeated students.Source: §B.2 · p. 28

B.3–B.4 Sensitivity analyses

  • After excluding students who completed both sections, combined AI–human equivalence and the Quantitative associations remained, while a gain-only clustered check did not establish equivalence in either section.Lesson-planning and practice-problem rankings remained unchanged, but the sensitivity results do not establish that every AI-tutor comparison is unchanged.Source: §B.3 · p. 28, §B.4 · p. 29

B.5 Comparing the human −AI gap between sections

  • 2.76 percentage points was the human-by-section interaction (95% HC3 CI [−2.38, 7.90]), leaving the difference between sections inconclusive.The selected highest AI means in Figure 1A may come from different tutors, and pooled-AI comparisons do not test those selected means.Source: §B.5 · p. 29

B.6 Sensitivity across student and tutor subsets

  • Sensitivity analyses across proficiency, background, timing, and tutor subsets use overlapping observations and do not establish equivalence in every subgroup.Source: §B.6 · p. 29

B.7 Exploratory equivalence and cost by AI tutor

  • Six of 12 individual AI tutors passed equivalence tests without correction; Gemma 4 31B’s AI −human difference was −1.09 percentage points (90% CI [−3.93, 1.75]; p=.044).Gemma 4 31B cost $0.00524 per percentage point versus $4.80508 for human tutoring, or 918× lower cost after rounding.Source: §B.7 · pp. 29–30

B.8 Separation across leaderboards; C.1–C.2 Learning gains and domain leaderboards

  • 53 of 55 AI-tutor pairs had nonoverlapping pointwise 95% intervals on at least one of five leaderboards, a descriptive comparison distinct from Holm-corrected tests.Section-specific gains adjust to a shared pre-test distribution; the highest AI mean exceeded the human mean in five domains, while human tutoring was higher in algebra and sentence equivalence.Source: §B.8 · p. 30, §C.1 · p. 30, §C.2 · p. 30

C.3 Starting proficiency and subject matter

Starting-proficiency analyses find stronger evidence of AI gains in some Quantitative score groups, while tutor differences vary by domain and proficiency jointly. The accompanying figures distinguish pooled comparisons from tutor- and domain-level patterns.

C.3 Starting proficiency and subject matter

  • AI tutoring gains exceeded control in the two lowest Quantitative pre-test groups (both p<.001), while the highest-group difference was inconclusive (p=.129).Quantitative score groups were ≤9, 10–13, 14–17, and ≥18 correct answers; Verbal groups used different cutoffs.Source: §C.3 · p. 31
  • Tutor differences varied jointly with domain and quartile in Quantitative (χ2 108=181.49, p<.001), but the Verbal interaction was inconclusive (χ2 72=83.16, p=.173).No within-domain–quartile test of tutor differences was significant, so the analyses do not establish a best tutor for any individual subgroup.Source: §C.3 · p. 31
  • Figure C.1 shows observed and pre-test-adjusted gains by section; adjusted gains put tutors on the same starting-score distribution within each section.Within each section, panels share a horizontal scale and are ordered by adjusted gain.Source: Figure C.1 · p. 31
  • Among the highest Quantitative pre-test quartile, tutor differences were inconclusive (285 students; p=.120), although Gemini 3.5 Flash had the highest mean gain at 15.8 percentage points.Gemini tutors occupied four of the five highest ranks in this exploratory comparison.Source: §C.3 · p. 32
  • Figure C.2 ranks tutors independently within each domain, so bar lengths should not be compared across panels; Table C.2 reports domain gains relative to control.Source: Table C.2 · p. 32, Figure C.2 · p. 33
  • Figure C.3 plots pooled AI–control gain differences across four pre-test groups; Figure C.4 shows individual-tutor differences by Quantitative domain and quartile.Each AI cell in Figure C.4 contains 12–35 students; positive values favor AI and negative values favor control.Source: Figure C.3 · p. 34, Figure C.4 · p. 34

D Tutoring cost, AI reply time and student engagement by section

This section examines AI tutor costs and frontier membership across GRE sections, exploratory links between tutoring-process measures and learning gains, and related expert-review, pedagogy, dispute, and prompt-design analyses. It also reports prompt-pilot comparisons, including descriptively higher gains with expanded prompts but inconclusive adjusted differences and no significant tutor-specific contrasts.

D Tutoring cost, AI reply time and student engagement by section

  • Four AI tutors had the lowest cost per percentage point of learning gain in Quantitative, Verbal, and Combined: Gemma 4 31B, GPT-5.4 mini, Kimi K2.6, and Gemini 3.7 Flash.Other rankings reversed by section: Gemini 3.5 Flash cost less than Gemini 3.6 Flash in Quantitative but more in Verbal; Opus 5 and Opus 4.8 (x-high) also reversed order.Source: D Tutoring cost, AI reply time and… · p. 35

B Verbal: learning gain vs. tutoring cost

  • Gemini 3.1 Pro was on the Verbal cost frontier but not Quantitative, while Gemini 3.5 Flash was on the Quantitative frontier but not Verbal.Human tutoring had the highest mean learning gain in Verbal, while several AI tutors had higher means in Quantitative; frontier membership uses sample estimates and may vary with sampling.Source: D Tutoring cost, AI reply time and… · p. 35, §D.1 · p. 36

D.2 Dialogue, practice and learning

  • Quantitative and Combined associations retained their directions under robustness checks, but some temporal comparisons were less consistent.After excluding repeat participants, all seven selected Combined associations persisted across 1,985 sessions; early engagement with late correct practice had p=.091 in Quantitative, and late correct practice with gain had p=1.00 in Verbal.Source: §D.2 · p. 39, Table D.2 · p. 39

E.2 Repeated expert reviewers

  • Expert review used eight shared criteria for lesson plans and practice-problem design; after accounting for repeated reviewers, Opus 5 no longer differed significantly from Opus 4.8 (off) in planning or overall ranking.The 2,028 reviews covered 852 comparison tasks; Opus 5 also no longer differed significantly from GPT-5.5 in practice design after the sensitivity adjustment.Source: §E.1 · pp. 39–40, §E.2 · p. 40

F Conversational pedagogy and student-reported problem quality

  • The conversational-pedagogy analysis measured six literature-derived tutoring rules, but did not validate them as predictors of learning lift.Human sessions were spoken video calls and AI sessions were typed chats, so the same rules may behave differently across modalities.Source: F Conversational pedagogy and… · p. 43

F.1 Student-reported answer disputes

  • 523 Quantitative answers and 120 Verbal answers were disputed; all observed Verbal rates were below one dispute per 100 answered problems, and tutor rates differed significantly in all sections (p<.001).A dispute records disagreement with a marked answer, not a verified error in the problem or answer key.Source: §F.1 · p. 45, Figure F.3 · p. 45

F.2 Agreement with expert judgments

  • Practice-design preference ranks correlated with fewer disputes in Quantitative (ρ=0.713, p=.035) and Combined (ρ=0.773, p=.033), but not Verbal (ρ=0.315, p=.388).For example accuracy, correlations were 0.895 (p<.001) in Quantitative, 0.406 (p=.388) in Verbal, and 0.782 (p=.033) in Combined; these exploratory tests treated estimated tutor ranks as fixed.Source: §F.2 · p. 45

Minimal and expanded prompts pilot

  • In a Quantitative pilot, minimal lesson-plan and tutor prompts produced a pooled mean gain of 14.27 percentage points, versus 8.85 with both prompts expanded.Mean gains were higher with minimal prompts for each AI tutor; pilot sessions informed the main-study prompts but were excluded from the main learning cohort.Source: Minimal and expanded prompts pilot · p. 46
  • None of the three tutor-specific expanded-versus-minimal prompt contrasts or their pairwise differences was significant.The exploratory pilot compared matched minimal and expanded settings across three AI tutors.Source: Minimal and expanded prompts pilot · p. 46, Figure G.1 · p. 47
  • 18.79 versus 15.31 percentage points for Opus 5 and 19.22 versus 14.65 for Gemini 3.6 Flash descriptively favored expanded prompts in later cohorts.After adjustment, the differences were 3.61 points (95% CI [−1.66, 8.89]; p=.359) and 4.62 points (95% CI [−0.17, 9.42]; p=.177); the interaction was inconclusive (p=.781). Cohorts used expanded prompts at different times from the main-study minimal-prompt cohorts, so prompt effects were not isolated.Source: Minimal and expanded prompts pilot · p. 46, Figure G.1 · p. 47
Loading 2609.28470v1…