Source-linked AI summary
LLM Prompt Evaluation for Educational Applications
Langdon Holmes, Adam Coscia, Scott Crossley, Joon Suh Choi, Wesley Morris
TL;DR
Educational applications need more systematic evidence for designing and evaluating LLM prompts. This study compares six pedagogically distinct prompt templates through a tournament of generated follow-up questions, finding that the Strategic Reading Coach substantially outperformed the alternatives. The framework is intended to support evidence-based prompt development while its specific effectiveness findings remain context- and model-bound.
Problem
Principled prompt evaluation is under-studied despite the growing use of LLMs in education and the sensitivity of performance to prompting strategies.
Method
The study compares six prompt templates for reading-dialogue follow-up questions using a tournament-style comparative judgment framework with Glicko2 ratings and an educational rubric.
Results
81% against the Scaffolding Expert and at least 90% against all other prompts, the Strategic Reading Coach had the highest estimated win probabilities in pairwise comparisons.
Takeaways & Limitations
The tournament provides quantitative evidence for distinguishing prompt templates and supports systematic prompt development beyond ad-hoc engineering.
Takeaways & Limitations
The effectiveness findings may not generalize beyond follow-up questions in this reading-comprehension context or beyond the single Llama 3 model evaluated.
Abstract
from arXiv · showhide
As large language models (LLMs) become increasingly common in educational applications, there is a growing need for evidence-based methods to design and evaluate LLM prompts that produce personalized and pedagogically aligned out-puts. This study presents a generalizable, systematic approach for evaluating prompts, demonstrated through an analysis of LLM-generated follow-up questions in a structured dialogue activity. Six prompt templates were designed and tested. The templates incorporated established prompt engineering patterns, with each prompt emphasizing distinct pedagogical strategies. The prompt templates were compared through a tournament-style evaluation framework that can be adapted for other educational applications. The tournament employed the Glicko2 rating system with eight judges evaluating question pairs across three dimensions: format, dialogue support, and appropriateness for learners. Data was sourced from 120 authentic user interactions across three distinct educational deployments. Results showed that a single prompt related to strategic reading out-performed other templates with win probabilities ranging from 81% to 100% in pairwise comparisons. This prompt combined persona and context manager pat-terns and was designed to support metacognitive learning strategies such as self-directed learning. The methodology showcases how educational technology re- searchers can systematically evaluate and improve prompt designs, moving beyond ad-hoc prompt engineering toward evidence-based prompt development for educational applications.
1. Introduction
LLMs are increasingly used in education, but prompt development and evaluation remain insufficiently systematic. This study introduces a tournament-style framework for comparing prompt templates in a reading-comprehension dialogue application.
- Few educational LLM studies describe a formal process for developing and evaluating prompts.
- Prompt templates provide customizable blueprints that separate fixed instructions from situation-specific inputs.
- Prompt patterns offer reusable solutions for recurring LLM prompt-engineering problems and can be combined into more sophisticated templates.
- Comparative judgment evaluates outputs by asking judges to choose between alternatives, supporting holistic assessment of complex output quality.
- The study applies systematic prompt evaluation to follow-up questions in a structured reading-comprehension dialogue system.
- STAIRS activates when a summary falls below a content threshold, selecting a less deeply engaged passage for review and generating a question with Llama 3 based on self-explanation reading training.
2. Methods
The study tested six theory-informed prompt templates on authentic reading-dialogue interactions and compared their generated follow-up questions through an adaptive tournament. Judges used a holistic rubric covering format, dialogue support, and appropriateness for adult learners.
- The study used authentic interactions from three educational deployments to populate prompts for evaluation.The data included economics, psychology, and programming textbook interactions.
- Six templates were evaluated: one existing Baseline and five templates designed with established prompt patterns and learning-theory principles.
- All templates received the same textbook, passage, initial SERT question, and learner response as variable inputs.Static and repeated prompt text was placed first to support prefix caching, while request-specific inputs were appended.
- The templates implemented distinct strategies, including Socratic questioning, cognitive verification, constructivist connections, strategic reading, and comprehension monitoring.
- Prompt Tournament: The tournament paired follow-up questions and used Glicko2 ratings to estimate rankings and Bradley-Terry win probabilities.
- Prompt Tournament: Eight judges selected preferred questions using a holistic rubric assessing format, dialogue support, and appropriateness for adult learners.The rubric emphasized direct questions, building on the dialogue, deeper thinking, respectful treatment, prior-knowledge connections, and metacognitive reflection.
3. Results
Prompt templates differed substantially in follow-up-question effectiveness. Strategic Reading Coach was strongest, while results also linked performance to particular combinations of prompt patterns and pedagogical features.
- Results indicated that learning-theory grounding produced more effective follow-up questions when combined with appropriate prompt-engineering patterns.
- Strategic Reading Coach achieved an estimated 81% win probability against Scaffolding Expert and at least 90% against all other templates.The tournament used adaptive comparisons, with matchup trial counts ranging from 0 to 91.
- Scaffolding Expert ranked second overall and achieved 100% against Baseline, 96% against Socratic Guide, and 91% against Connection Builder.
- The Baseline template ranked third overall despite lacking explicitly designed prompt patterns.Its emphasis on Bloom's Taxonomy and metacognition aligned with the pedagogical aims of the follow-up question.
- Strategic Reading Coach combined Persona and Context Manager patterns, while Scaffolding Expert combined Cognitive Verifier and Reflection patterns.These combinations were associated with consistently strong results and effective structured analysis before question generation.
4. Discussion
The study finds that systematic prompt evaluation can identify meaningful differences among educational prompt templates. SRC performed best, while results also show that empirical testing matters because theoretical alignment alone did not guarantee effective performance, and the findings remain context- and model-specific.
- SRC won every pairwise comparison, with win probabilities ranging from 81% to 100%.Its combination of Persona and Context Manager patterns supported pedagogically focused follow-up questions that adapted to learner responses.
- The Baseline template placed third despite lacking explicitly modern prompt patterns, while the top two templates showed a substantial performance gap over it.The authors relate this result to the continued effectiveness of metacognition and Bloom’s Taxonomy, alongside gains from thoughtful prompt-pattern application.
- The poor performance of Connection Builder indicates that theoretical alignment alone does not guarantee effective prompt performance.The finding underscores the importance of empirical evaluation in prompt engineering.
- The tournament methodology provides a systematic alternative to ad-hoc prompt engineering and can identify meaningful differences between templates.The study presents the approach as adaptable to other educational applications, while its specific effectiveness findings are bounded by the evaluation setting.
- The evaluation focused on follow-up questions in reading comprehension and used only Llama 3, so specific prompt-effectiveness findings may not generalize across contexts or models.The authors call for cross-model evaluation because different models may respond differently to the same prompt patterns.
5. Future Work
Future work should adapt the tournament approach across educational tasks, contexts, pedagogical goals, prompt-pattern combinations, and LLMs.
- Future research should test prompt-pattern and LLM combinations across varied educational tasks, contexts, and pedagogical goals.The tournament approach is presented as adaptable beyond follow-up-question generation in reading comprehension.
Declaration on Generative AI
The authors report that they did not use Generative AI tools.
- The author(s) have not employed any Generative AI tools.