Effective Personalized AI Tutors via LLM-Guided Reinforcement Learning

Angel Tsai-Hsuan Chung; Botong Zhang; Ling-Chieh Kung; Hamsa Bastani; Osbert Bastani

Source-linked AI summary

Effective Personalized AI Tutors via LLM-Guided Reinforcement Learning

Angel Tsai-Hsuan Chung, Botong Zhang, Ling-Chieh Kung, Hamsa Bastani, Osbert Bastani

arXiv:2608.16907v1Jul 10, 2026cs.CYcs.AI

TL;DR

Existing GenAI tutors mainly react to questions and use coarse indicators of student understanding. This paper combines a GenAI chatbot with reinforcement learning to adaptively sequence practice problems, increasing unassisted final-exam performance by 0.15 standard deviations.

Problem
Tutoring systems often rely on coarse proxies for student knowledge, providing an incomplete approximation of student understanding.
Method
The platform uses reinforcement learning to sequence practice problems from richer signals in student solutions and chatbot interactions.
Results
0.15 standard deviations: adaptive sequencing improved unassisted end-of-semester exam performance, with gains linked to increased student engagement.
Takeaways & Limitations
The study provides field evidence that GenAI tutors can improve educational outcomes through proactive, personalized learning support.
Takeaways & Limitations
The evaluation focused on introductory Python learning among high school students.

Abstract

from arXiv · show

Generative AI (GenAI) is rapidly reshaping education by unlocking the potential for personalized tutoring. Yet, emerging platforms largely focus on GenAI chatbot tutors that reactively answer student questions. We hypothesize that the efficacy of GenAI chatbot tutors can be substantially improved by proactively guiding student learning. To test this, we design a novel tutoring platform that tightly integrates a carefully-designed GenAI chatbot with a reinforcement learning algorithm for sequencing practice problems. Critically, this algorithm leverages rich signals from student-chatbot interactions to adaptively select practice problems of an appropriate difficulty level. In partnership with the Taipei City Government and American Institute in Taiwan, we deployed our tutoring platform in conjunction with a five-month course to teach Python to students across ten high schools. We randomized students between a fixed practice problem sequence and our adaptive sequencing algorithm. We find that adaptive sequencing increased unassisted final exam performance by 0.15 standard deviations (equivalent to 6-9 months of schooling by some estimates); mediation analysis suggests that gains were driven by increased engagement. Our work provides large-scale field evidence that student-chatbot interactions provide valuable signals for proactively optimizing and personalizing student learning.

1 Introduction

The study integrates a GenAI chatbot tutor with reinforcement-learning-based adaptive problem sequencing that uses rich student-platform signals to personalize difficulty. In a randomized trial across 10 Taipei high schools, personalized sequencing improved unassisted exam performance by 0.15 standard deviations, with gains mediated by engagement.

Motivation: GenAI chatbot tutors are primarily reactive, while effective learning requires challenging practice and students often lack the self-regulation to formulate beneficial queries.The introduction identifies productive struggle and difficulty-appropriate practice as important, but notes that chatbot users may struggle to direct their own learning.
Problem: Existing adaptive algorithms rely on coarse signals such as correctness and time, limiting estimation of students’ rich knowledge states.Bayesian knowledge tracing is presented as an example that maps binary solution outcomes to a binary learned-or-unlearned state.
Method: The proposed reinforcement-learning algorithm combines LLM-extracted semantic and chatbot-interaction signals with particle filtering and model predictive control to estimate knowledge and personalize problem difficulty.The system uses these signals to form a continuous knowledge-state measure and select practice problems at granular difficulty levels.
Experiment: The five-month course randomized 770 students across 10 Taipei high schools to personalized problem sequencing or a fixed sequence, with the same course material and chatbot tutor.The randomized controlled trial augmented standard lectures with tutoring-platform practice.
Contribution: Rich behavioral signals from student-platform interactions enabled automated personalization that substantially improved learning outcomes in a large-scale field deployment.The study characterizes its evidence as rigorous field evidence and contrasts its scale and horizon with existing GenAI tutoring work.
Results: 0.15 standard deviations: personalized sequencing improved performance on an in-person written exam completed without AI assistance versus fixed sequencing.The improvement corresponds to as much as 6–9 months of additional schooling according to some estimates, without increasing instruction time or teacher workload; mediation analysis suggests increased engagement drove the gains.

2 GenAI Tutoring System

The GenAI tutoring system combines structured practice, an LLM chatbot, and adaptive problem sequencing informed by rich student-interaction traces. Its workflow generates and validates course-aligned problems, supports iterative coding practice, and estimates mastery to select among four difficulty levels.

System overview: The platform organizes lecture instruction and practice problems into ordered modules, while the system generates problem banks, supports problem solving, and selects problem sequences to maintain engagement.These three functions are available across the tutoring platform, with sequencing access randomized in the experimental design.
Learning signals: The system uses student-tutor interactions and code-edit traces to estimate learning progress, providing a more precise proxy than conventional measures.Recorded traces include chatbot messages, code edits, and submissions, with examples distinguishing answer-seeking from concept-understanding engagement and superficial from meaningful edits.
Problem generation: LLM-generated candidate problems are aligned with course materials through retrieval-augmented generation and reviewed by teaching assistants before entering the problem bank.The generator produces auto-gradable problems, while human review checks quality, correctness, language clarity, and course alignment.
Student interface: The web interface combines sequential practice, code execution, an autograder, and a chatbot that answers questions, offers hints, and interprets error messages.Students receive immediate feedback and may submit solutions repeatedly without penalty, while the tutor is prompted to withhold answers until substantial effort is shown.
Adaptive sequencing: The adaptive sequencing algorithm models mastery continuously in a POMDP and selects the next problem’s difficulty out of four levels using the estimated knowledge state.Its knowledge state represents initial performance, mastery ceiling, and learning speed, addressing the inadequacy of binary feedback when students make errors before solving correctly.
Adaptive sequencing: The method’s mastery estimate can improve rapidly, enabling strong performance earlier in the problem sequence.The knowledge state evolves based on learning speed, attempts, and selected difficulty, with transition-function hyperparameters calibrated using pilot-study data.

3 Field Experiment

The field experiment evaluated personalized problem sequencing in a five-month, government-launched Python program across ten Taipei high schools. Students were randomized to adaptive or fixed problem sequences, with identical instructional content and chatbot support, and performance measured on an unassisted in-person certification exam.

Participants and program: 1,047 students from 10 Taipei high schools participated, with results reported for 770 students after preregistered exclusions.The schools included 8 public and 2 private institutions.
Participants and program: The program ran from January through June 2025 for approximately five months, ending with an in-person certification examination scored by external graders.The government officially launched the program in January 2025, and the exam was administered in June 2025.
Experimental design: Students were randomized individually with equal probability to adaptive POMDP-based problem sequencing or a fixed easy-to-hard sequence.The treatment group received personalized problem sequences, while the control group followed a fixed sequence within each module.
Experimental design: The learning platform, instructional content, and tutor chatbot were identical across groups, differing only in problem sequence.All students also completed at least two very hard, two hard, and two medium questions in each module.
Outcome measurement: The primary outcome was performance on an in-person final certification exam without digital devices, assessing conceptual understanding and Python problem-solving.Trained graders scored all exams using a common rubric.

4 Evaluation

Personalized problem sequencing improved standardized exam performance, with gains concentrated among beginners and lower-tier schools. The evidence indicates that increased engagement and higher-quality problem-solving, rather than solving more or uniformly harder problems, drove the improvement.

Overall effects: 0.156 standard deviations: treatment-group students outperformed controls on certification exam performance (p < 0.05).With baseline covariates, school fixed effects, and grade-level fixed effects, the gain was 0.150 standard deviations (p < 0.05), equivalent to approximately six to nine months of additional schooling.
Treatment heterogeneity: 0.215 SD (p = 0.012): personalization increased performance for beginners, versus 0.008 SD (p = 0.941) for students with prior Python skill.Beginners comprised 53% of students, while those with prior Python skill comprised 39%.
Treatment heterogeneity: 0.173 SD (p = 0.045): gains were larger in lower-tier schools, compared with 0.039 SD (p = 0.752) in higher-tier schools.Estimated effects were positive for both school types, and the overall pattern suggests gains without widening educational inequality.
Mechanisms: 0.185 SD via totalTime and 0.149 SD via totalAttempt: both engagement measures showed positive indirect effects (p < 0.001).Direct effects were small and statistically insignificant, implying that the approximately 0.15 SD total effect was almost fully mediated by increased engagement.
Mechanisms: Nearly identical numbers of completed questions: treatment and control groups differed in practice quality rather than practice quantity.Although treatment students encountered harder questions on average, increased engagement—not solving more or harder problems—was identified as the main driver of the effect.

5 Conclusion

The study shows that integrating a chatbot tutor with reinforcement-learning problem sequencing can personalize learning by using behavioral traces, improving exam performance through greater engagement without increasing instructional demands. The authors suggest this approach could support effective personalized learning and reskilling at scale.

System contribution: The tutoring system combines a GenAI chatbot with reinforcement learning for personalized problem sequencing.The algorithm uses code edits and student-chatbot conversations as rich signals of student mastery rather than coarse performance proxies.
Evaluation: 770 high school students participated in a randomized controlled trial evaluating the system.The evaluation tested the integrated tutoring system in a field setting.
Results: 0.15 standard deviations improved end-of-semester exam performance, equivalent to six to nine months of additional schooling by some estimates.These gains occurred without increasing instruction time or teacher workload.
Results: Increased student engagement with practice problems drove the performance gains.The conclusion emphasizes personalization of the problem sequence rather than personalization of the tutor chatbot.
Broader implications: Although evaluated in introductory Python learning among high school students, tightly integrated task sequencing and chatbot tutoring may enable personalized learning and reskilling at scale.The authors relate this potential to educational settings and workplace reskilling that share heterogeneous proficiency, steep learning curves, and productive struggle.

End Notes

The paper acknowledges institutional, school, technical, administrative, and research support for the study. It also reports funding, author contributions, no competing interests, and ethics and inclusion procedures.

Acknowledgments: The authors acknowledge partnerships with the Taipei City Government, American Institute in Taiwan, American Innovation Center, Better Together for NextGen Taiwan, and participating-school teachers.Teachers played a critical role in program implementation.
Funding: The research was funded by the Wharton AI & Analytics Initiative, Wharton Mack Institute for Innovation Management, an Amazon Research Award, Taipei City Government, and American Institute in Taiwan.The Amazon Research Award was from Fall 2023.
Author contributions: HB, OB, and AC contributed to study design and methodology; HB, OB, AC, and BZ wrote the manuscript; and AC and BZ analyzed results under HB and OB’s supervision.AC and LK contributed to study implementation.
Competing interests: The authors declare that they have no competing interests.
Ethics & Inclusion Statement: The study received University of Pennsylvania IRB approval under Protocol # 855951 and exempt status under Category 1, with formal partnership and data-privacy alignment in Taiwan.The team worked with policymakers and frontline personnel, conducted training sessions, and aligned the approach with government systems and priorities.

Data and materials availability: … A.2 RAG System for Problem Generation

The supplementary materials describe a personalized AI learning platform that combines lectures, practice, and an AI tutor with adaptive problem sequencing, alongside an RAG pipeline for generating bilingual Python questions. Data, code, supporting analyses, and course materials are available for further research and replication.

Data and materials availability:: Data and code will be released through a GitHub repository upon publication, with replication materials already accessible online.The current replication link is https://tinyurl.com/yx7vsrtc.
Data and materials availability:: The supplementary materials cover the personalized learning system, RL algorithm, pilot study, experimental protocol, data processing, additional analyses, related work, and course materials.Course-related materials include example practice problems and the certification exam.
A Personalized AI Learning System: The personalized web platform integrates lecture videos, coding practice, and AI tutoring in one interface, with modules released sequentially while allowing within-module personalization.The platform is integrated with Taipei’s government e-learning system and uses government-authenticated accounts linked to academic records.
A.1 Web Platform: The web platform presents lecture content, example problems, an always-available LLM tutor, and browser-based practice with immediate feedback and several question types.The tutor provides clarification, hints, and error explanations while avoiding direct answers until students demonstrate effort; practice includes coding, debugging, and true/false questions.
A.1 Web Platform: Control students follow a fixed easy-to-hard sequence, whereas treatment students receive the next question from an RL policy using an inferred belief state.Each module contains 40 potential questions, with homework typically requiring 15 and experimental conditions requiring minimum exposure to medium, hard, and very hard questions.
A.1 Web Platform: The platform records time-stamped activity logs, including code edits, submissions, navigation, practice outcomes, and conversations with the AI tutor.These records support both construction of the learner model for the RL policy and empirical analyses of learning behavior and outcomes.
A.2 RAG System for Problem Generation: The RAG-based LLM pipeline generates Python questions from instructional content, optional additional materials, and existing questions used to discourage duplicates.Each request produces two coding questions in English with traditional Chinese translations, plus solution stubs, sample solutions, and assert-based test suites.
A.2 RAG System for Problem Generation: Generation prompts specify educational content, output constraints, difficulty levels, bilingual statements, and test-suite requirements, with randomized sampling to encourage variety.The sampling settings are temperature = 1.0, top-p = 0.7, and top-k = 200, and questions are generated in batches of two.

A.3 GenAI Chatbot Tutor … B.3.1 Experimental Setup

The platform combines a GPT-4o chatbot tutor with model-based reinforcement learning that sequences practice difficulty using latent-state estimation and LLM-derived signals. The paper evaluates this approach through a pilot-data simulation with 37 students split into training and test sets.

A.3 GenAI Chatbot Tutor: The GPT-4o tutor was embedded in practice for both RL and fixed-sequencing conditions, with shared context, concise middle-school explanations, targeted hints, and no direct answers.The tutoring model and prompting pipeline were held constant across sequencing conditions, while pedagogy prompts were additionally randomized.
A.3 GenAI Chatbot Tutor: Pedagogy randomization produced no meaningful or statistically significant differences in exam performance or engagement, and the reported results were robust to it.All pedagogy prompts prohibited direct answers or full code, required concise responses, and instructed diagnosis of formulation versus Python difficulties.
B Reinforcement Learning for Problem Sequencing; B.1 Partially Observable Markov Decision Process (POMDP); B.2 Model-Based Reinforcement Learning: The sequencing algorithm models each student-module pair as a POMDP, predicts its parameters from historical data, and uses optimal control to select practice problems.LLMs analyze code submissions and student-tutor interactions to construct observations and machine-learning features; simulation evidence supports their value.
B.1.1 Overview; B.1.2 Latent States, Actions, Transitions, and Observations; B.1.3 Belief State and Belief State Estimation: The POMDP selects Easy/Medium/Hard/Very Hard difficulty from a belief state and updates uncertainty through Bayesian filtering after noisy student interactions.Its latent state represents learner effort, while observations summarize effort and performance; LLMs identify meaningful code attempts rather than superficial resubmissions.
B.1.2 Latent States, Actions, Transitions, and Observations: The model represents learners with initial mastery, asymptotic mastery, and learning speed, using Poisson-distributed meaningful attempts as noisy effort observations over a 40-step horizon.Harder tasks tend to induce larger observed effort, and lower effort indicates higher proficiency.
B.1.4 Reward Function: The reward favors difficulty matching the learner’s estimated skill, increasing difficulty according to personal learning speed while accommodating fast and slow learners.It gives reward 1 for an exact match and penalizes deviations linearly; progression rates were expert-specified and calibrated with pilot trajectories.
B.2 Model-Based Reinforcement Learning; B.3 Simulation Experiment: The GBM predicts whether a learner is slow or fast from behavioral logs, code histories, difficulty and time-to-solve summaries, meaningful attempts, and chatbot interaction quality.Pilot evaluation found GBMs substantially outperformed other model families, and the simulation used randomized problem sequences to assess decision-making.

B.4 Simulation Results

The POMDP policy with ML-based initialization achieved the highest simulation reward, while LLM-derived features improved prediction of student learning speed and meaningful attempts explained more learning-outcome variation than raw attempts.

Quantitative results: 0.942 overall average reward was achieved by the POMDP-based policy with ML-based initialization, with rewards of 0.975 for high τ and 0.903 for low τ students.The policy performed consistently across both student groups.
Quantitative results: 0.935 overall performance for ML no LLM was slightly below ML, indicating that LLM-derived features contributed to improved adaptation.Greedy achieved 0.931 overall, while PPO achieved 0.898 overall with balanced rewards of 0.904 versus 0.890 across high- and low-τ groups.
Quantitative results: 0.843 overall performance for the No historical baseline, including 0.821 for high τ students, was substantially below the 0.926 Uniform baseline, while Random and Fixed achieved 0.499 and 0.578.The results indicate that historical data and adaptive sequencing are important for performance.
Prediction model evaluation: 72.44% AUC was achieved when LLM features were included, compared with 63.66% without them, with meaningful edits and chat-quality features capturing engagement and help-seeking behavior.Several LLM-derived features also had high importance across multiple evaluation metrics.
Efficacy of LLM-based observations: R2 = 0.14 was explained by the meaningful-attempt model, compared with R2 = 0.08 for the raw-attempt model.Adding meaningful attempts produced F = 67.36, p < 0.001 improvement in fit, whereas adding raw attempts produced F = 15.66, p < 0.001.
Efficacy of LLM-based observations: 69.06% of explained variance was attributed to zMeaningfulAttempt, versus 30.94% for zTotalAttempt, confirming that meaningful attempts contributed most of the joint model’s explanatory power.The analysis cautions that coefficient magnitudes are difficult to interpret because the predictors share variance.

C Pilot Study … D.3 Survey Instrument

The paper first used an IRB-approved pilot to validate and redesign the AI learning system, while the main protocol specified teacher briefings, certification procedures, and a baseline survey. The pilot informed offline RL development but was not treated as a formal test of the fully functional adaptive algorithm.

C Pilot Study: The IRB-approved pilot assessed end-to-end system feasibility, collected learner trajectories for RL calibration, and incorporated feedback into platform and question-bank redesign.The redesign used quantitative usage traces and structured feedback from students and teaching staff.
C Pilot Study: 52 students participated in the two-month 2024 NTU summer pilot, including high school or first-year college students, graduate students, and participants from multiple universities.Twenty-two students studied outside Taipei, and human graders assessed in-person midterm and final exams.
C Pilot Study: Data infrastructure and integration problems prevented reliable online RL-policy updates, so the pilot primarily supported system validation and data collection.Control-arm interaction logs were used to train and calibrate RL components offline and guide comprehensive re-implementation.
D Experimental Protocol Details: The experimental-protocol details cover teacher and student briefings, the certification-program schedule and rules, survey items, and preregistered analyses.These subsections organize the implementation and measurement procedures described in the appendix.
D.1 Teacher and Student Briefing: Teachers received in-person protocol and platform briefings, recruited consenting students, and supervised implementation while students remained blinded to specific treatment assignments.The enrolled sample consisted primarily of 10th-graders, with reported distributions of prior Python experience.
D.2 Certification Program Schedule and Rules: The certification program used multi-week online modules with videos, practice, an embedded AI tutor, assignment deadlines, and an on-site paper-based assessment.Students received login credentials, orientation materials, AI-tutor usage guidance, and academic-integrity rules.
D.3 Survey Instrument: A baseline survey administered after the introductory module measured demographics, motivation, parental education, and familiarity with course material.The survey was conducted after students became familiar with the platform, and its questions are listed in Table 5.

D.4 Pre-Registration Details … E.2 Balance Table

The paper preregistered primary treatment-effect and pedagogical-strategy analyses, reported only the high-school deployment, and used prespecified exclusions to form a balanced final analysis sample of 770 students.

D.4 Pre-Registration Details: The primary outcome was student performance on the certification exam, analyzed using preregistered regression and pedagogical-strategy evaluations.The regression controlled for baseline covariates and fixed effects.
D.4 Pre-Registration Details: The study found no meaningful variation across prompt conditions and no meaningful or statistically significant differences across the three pedagogical frameworks.The frameworks were Constructivism, Behaviorism, and Social Learning.
D.4 Pre-Registration Details: Secondary preregistered analyses examined heterogeneous treatment effects by prior Python experience and school tier, plus engagement patterns and AI chat conversations as potential mechanisms.These analyses used platform logs and AI chat conversations.
D.4 Pre-Registration Details: Only the high-school deployment was reported because unforeseen delays prevented reliable intervention delivery during the NTU university deployment.The paper also reports that main results remain robust when controlling for pedagogical prompt features.
E Data Processing, Attrition, and Covariate Balance: Data processing, attrition, and covariate balance were addressed through sample exclusion criteria and treatment-arm balance assessments.Section E.1 details exclusions, while E.2 examines covariate balance across arms.
E.1 Exclusion Processing: 1,047 students participated, and prespecified processing excluded 94 dropouts or exam absences, 29 students flagged for cheating, and 154 senior students affected by delayed RL implementation.The senior-student RL algorithm became active one month after course start, about half their accelerated course period.
E.2 Balance Table: 770 students remained in the final analysis sample, with no evidence of systematic differences in observed pretreatment characteristics across treatment conditions.Table 7 reports covariate balance between treatment and control groups.

E.3 Differential attrition · F Evaluation

Attrition was balanced across treatment arms and statistically unrelated to assignment, reducing concern that sample exclusions bias estimated effects. The evaluation section also outlines robustness, alternative-explanation, heterogeneity, mediation, and feedback analyses.

E.3 Differential attrition: 26.7% of control students and 26.2% of treatment students were excluded from the main analysis sample.Exclusions were 140/524 in control and 137/523 in treatment.
E.3 Differential attrition: Attrition rates did not significantly differ by treatment status in either the two-sample proportion test (p = 0.85) or chi-squared test (p = 0.90).Both tests failed to reject equality of attrition rates.
E.3 Differential attrition: The treatment effect on attrition was statistically indistinguishable from zero in both school-fixed-effects specifications.The Logit specification estimated β = −0.057 (p = 0.73), while the LPM estimated β = −0.78 percentage points (p = 0.74).
E.3 Differential attrition: Overall, the results indicate negligible differential attrition between treatment and control, alleviating concerns that exclusions bias estimated treatment effects.The analysis defines attrition as exclusion from the main analysis sample.
F Evaluation: F.1 details regression specifications for the main analysis and robustness checks.The evaluation section is organized into subsections covering specifications, alternative explanations, heterogeneity, mediation, and feedback.
F Evaluation: F.2 evaluates whether increased problem difficulty could explain the findings, while F.3 presents heterogeneity analyses.These subsections address an alternative explanation and variation in effects.
F Evaluation: F.4 reports mediation analyses, and F.5 summarizes student feedback on the AI tutoring program.Together, these subsections examine mechanisms and participant responses.

F.1 Main Analysis and Robustness Checks · F.2 Ruling Out “Harder is Better”

The main analysis estimates RL personalization effects with OLS and finds that positive exam-performance effects remain under alternative standard errors, expanded-sample checks, and controls for randomized pedagogical prompts. Pilot evidence further indicates that increased difficulty alone does not explain the gains, which are instead attributed to adaptive sequencing rather than mechanical difficulty inflation.

F.1 Main Analysis and Robustness Checks: OLS regressions estimate RL personalization effects on raw and standardized final-exam performance, with specifications that progressively add baseline covariates and school and grade-level fixed effects.Standardized exam scores are computed as full-sample z-scores; covariates include prior programming experience, parental education, prior Python ability, gender, and grade-level fixed effects.
F.1 Main Analysis and Robustness Checks: The positive effect of RL personalization remains statistically significant across all three alternative-standard-error specifications.These calculations address potential heteroskedasticity and within-school correlation of residuals.
F.1 Main Analysis and Robustness Checks: Including senior students produces directionally similar but statistically insignificant estimates, consistent with their RL algorithm operating for only about half of the accelerated course period.With senior interactions, the main RL coefficient remains positive while the RL×Senior interaction is negative and statistically insignificant.
F.1 Main Analysis and Robustness Checks: 0.145 standard deviations (p = 0.026) is the estimated improvement in standardized exam scores after controlling for randomized AI-tutor pedagogical prompts.The randomized prompt frameworks were Constructivism, Behaviorism, and Social Learning; including them leaves the estimated RL effect essentially unchanged.
F.2 Ruling Out “Harder is Better”: The treatment group received somewhat higher-difficulty questions on average, creating a potential “Harder is Better” explanation that the main-study RCT cannot isolate because difficulty assignment was endogenous.Difficulty was dynamically determined by the RL policy in treatment and followed a fixed, increasing sequence in control, confounding harder exposure with student persistence.
F.2 Ruling Out “Harder is Better”: The pilot control arm identifies the causal effect of harder questions by randomizing difficulty and relating assigned difficulty to outcomes while controlling for baseline covariates and AI-tutor assignment.This design avoids the main-study limitation because difficulty is assigned exogenously in the pilot control arm.
F.2 Ruling Out “Harder is Better”: Increased difficulty alone does not account for improved performance and could potentially harm beginners, supporting adaptive sequencing rather than mechanical difficulty inflation as the explanation for the main-study gains.The pilot analysis uses AvgDif, defined as average assigned practice-question difficulty, and examines standardized exam performance.
F.2 Ruling Out “Harder is Better”: The main-study gains are driven by adaptive sequencing rather than by mechanical difficulty inflation.This conclusion follows from the pilot evidence that increased difficulty alone does not explain improved performance.

F.3 Heterogeneity Analysis · F.4 Mediation Analysis · F.5 Student Feedback

RL personalization benefited Python beginners most, with no evidence of disproportionate gains for higher-tier schools. Mediation analyses indicate that improved exam performance operated primarily through increased student engagement, while feedback was broadly positive despite early platform instability.

F.3 Heterogeneity Analysis: RL gains were largest among Python beginners, while the incremental effect for Python-familiar students was −0.249 SD (p = 0.066).The treatment effect for Python-familiar students was much smaller and not statistically distinguishable from zero.
F.3 Heterogeneity Analysis: −0.145 SD (p = 0.328) was the RL×SchoolTier interaction, providing no evidence that gains disproportionately favored higher-tier schools.Performance gains therefore did not systematically favor already advantaged students.
F.4 Mediation Analysis: ≈0.15 SD was the overall treatment effect, which operated primarily through increased engagement rather than a residual direct channel.The mediation analysis used total platform time and total attempts as engagement mediators.
F.4 Mediation Analysis: 0.193 SD (p < 0.001) was the ACME for totalTime, while its ADE was −0.031 SD (p = 0.682).For totalAttempt, the ACME was 0.152 SD (p < 0.001) and the ADE was 0.009 SD (p = 0.854).
F.4 Mediation Analysis: The direct effects were statistically indistinguishable from zero: βdirect (time) = −0.081 (p = 0.235) and βdirect (attempts) = −0.011 (p = 0.869).This pattern suggests full mediation through sustained student effort and extended productive practice.
F.5 Student Feedback: Students responded positively to the platform’s core learning experience, citing clear instruction, diverse practice materials, and a meaningful learning process.Representative feedback described the platform as easy to use and the explanations as very clear.
F.5 Student Feedback: Early lag and session timeouts arose from traffic congestion in the government system’s SSO linkage, but the issues were resolved and did not persist.The team fixed technical glitches and extended the first required homework deadline.

G Summary of Related Work on Adaptive Learning Policies … 證書測驗 Certification Exam

The paper reviews real-world adaptive learning studies and positions its large-scale, long-term institutional deployment as addressing limitations in prior STEM evidence. It then describes practice materials and a bilingual, handwritten certification exam spanning conceptual knowledge, code tracing, and programming tasks.

G Summary of Related Work on Adaptive Learning Policies: The review selected real-world STEM studies using adaptive algorithms to sequence or select content, comparing treatment and control learning outcomes while excluding pedagogy-only, simulated, and offline-only evaluations.The search combined literature review, frontier-LLM research, and manual verification against strict inclusion criteria.
G Summary of Related Work on Adaptive Learning Policies: Evidence is mixed for adaptive STEM sequencing: significant effects may involve added supports, weak baselines, or outcomes not directly tied to learning, while many studies lack institutional context.The two closely aligned large-scale field RCTs were Eedi and Doodle; Doodle reported no significant improvement in math scores.
G Summary of Related Work on Adaptive Learning Policies: The study claims the first statistically significant educational impact of mastery-based adaptive learning over expert-informed fixed-sequence practice in a large-scale, long-term, institutionally grounded deployment.The comparison specifically concerns practice-problem sequencing rather than overall platform efficacy.
H.1 Practice Problem Examples: Each module contains 40 AI-generated homework questions, with 10 questions at each of four difficulty levels, plus 10 backup questions per module and approximately 500 questions overall.Questions passing automated RAG validation were manually checked by six independent reviewers for accuracy and language clarity.
H.2 Certification Exam: The certification exam was manually designed from course materials and prior-term evaluation data, administered bilingually in person as a handwritten, paper-based, no-device assessment.The exam was closed book and lasted 120 minutes, with a total score of 100 points.
證書測驗 Certification Exam: A further 28-point programming problem required computing total tardiness for a specified task sequence from processing times and deadlines.In the worked example, the sequence (1, 4, 2, 3) produced 10 h of total tardiness.