Source-linked AI summary
Learning gain differences between ChatGPT and human tutor generated algebra hints
Zachary A. Pardos, Shreya Bhandari
TL;DR
The paper asks whether ChatGPT can generate algebra hints that support learning as effectively as human-authored hints, addressing the labor-intensive production of tutoring content. It compares both hint types in a 77-participant experiment across Elementary and Intermediate Algebra and finds stronger learning gains for human-authored hints, while identifying quality-control and design limitations.
Problem
The paper examines whether ChatGPT can produce educational hints comparable to human-authored hints, which matters because authoring tutoring content is labor intensive.
Method
The study compares ChatGPT-generated and human-authored hints in an online experiment with 77 participants across Elementary and Intermediate Algebra lessons.
Results
Human-created hints produced higher learning gains than ChatGPT hints in both lessons, with statistically significant gains only for the manual hint conditions.
Takeaways & Limitations
ChatGPT hint generation showed potential for algebra learning but still required human supervision and did not match the learning gains of manual hints in this study.
Takeaways & Limitations
The comparison may confound hint-authoring quality with differences between worked solutions and hint or scaffold formats, while immediate feedback also contributes to learning gains.
Abstract
from arXiv · showhide
Large Language Models (LLMs), such as ChatGPT, are quickly advancing AI to the frontiers of practical consumer use and leading industries to re-evaluate how they allocate resources for content production. Authoring of open educational resources and hint content within adaptive tutoring systems is labor intensive. Should LLMs like ChatGPT produce educational content on par with human-authored content, the implications would be significant for further scaling of computer tutoring system approaches. In this paper, we conduct the first learning gain evaluation of ChatGPT by comparing the efficacy of its hints with hints authored by human tutors with 77 participants across two algebra topic areas, Elementary Algebra and Intermediate Algebra. We find that 70% of hints produced by ChatGPT passed our manual quality checks and that both human and ChatGPT conditions produced positive learning gains. However, gains were only statistically significant for human tutor created hints. Learning gains from human-created hints were substantially and statistically significantly higher than ChatGPT hints in both topic areas, though ChatGPT participants in the Intermediate Algebra experiment were near ceiling and not even with the control at pre-test. We discuss the limitations of our study and suggest several future directions for the field. Problem and hint content used in the experiment is provided for replicability.
1 INTRODUCTION
The paper investigates whether ChatGPT can generate algebra hints that support learning and compare with human-authored hints. It frames automated hint generation as a possible response to the labor-intensive authoring of tutoring content.
- Motivation: Worked-solution hints have previously been associated with learning gains in algebra tutoring systems.The introduction cites evidence from secondary students and Mechanical Turk workers.
- Experimental comparison: The experiment randomly assigns participants to manual or ChatGPT hints and to Elementary or Intermediate Algebra lessons.Human-authored hints serve as the control, while ChatGPT-generated hints replace them in the experiment condition.
- Research questions: The study asks whether ChatGPT hints are low quality, produce learning gains, and compare with human tutor hints.These questions are evaluated using an online experiment with 77 participants across two algebra lessons.
- Motivation: Automated hint generation could help scale computer tutoring because creating educational and hint content remains labor intensive.The authors provide the tutor code and experiment content for reproducibility.
2 RELATED WORK
The related work surveys automatic hint generation across tutoring and programming domains, alongside the emergence of Transformer-based large language models. Prior approaches use observed solution paths, canonicalized states, policies, or expert-like heuristics to generate guidance.
- Large Language Models: Earlier work had evaluated GPT-3 for code explanations and language models for generating variations of math word problems.The paper positions its learning-gain study within this emerging educational use of language models.
- Automatic hint generation: Prior automatic hint-generation systems have modeled successful solution paths to guide future learners and reported positive learning outcomes.One logic-tutor approach represented student paths as a Markov Decision Process and demonstrated positive outcomes in practice.
- Programming education: Programming education research has explored state mappings, data-driven policy models, and expert-mimicking heuristics for generating hints.These approaches include verbatim or canonicalized solution states and comparisons between Poisson and Markov models.
- Large Language Models: Large Language Models are highly parameterized neural networks trained on large text corpora and commonly built on the Transformer architecture.The Transformer introduced attention mechanisms used to infer word meaning from sentence context.
3 METHODS
The methods select text-based algebra lessons, generate ChatGPT responses directly from problem prompts, and quality-check the resulting hints before study use. Human-authored OATutor hints provide the comparison condition.
- 3.1 Subject lesson selection: The study selects one Elementary Algebra lesson and one Intermediate Algebra lesson from OpenStax content available in OATutor.Algebra was chosen because it has extensive tutoring baselines and pre-authored questions and hints.
- 3.1 Subject lesson selection: Each selected objective requires at least eight problems, with no problems depending on images or figures.The design uses three pre-test items, five acquisition items, and repeated post-test items.
- 3.2 ChatGPT Hint Generation: ChatGPT is a chat interface to a machine-learning model based on the Generative Pre-trained Transformer architecture.It returns text in response to a user-provided prompt.
- 3.2 ChatGPT Hint Generation: For every problem, ChatGPT receives a concatenated text prompt containing the OATutor problem and step components in a new chat.The response is recorded as a potential hint.
- 3.2.3 Quality checks.: ChatGPT hints undergo checks for the correct answer, correct work, and absence of inappropriate language before use.The checks address the possibility that language models produce plausible but factually incorrect statements.
- 3.2.2 Manual hint generation: Human hints were authored by undergraduate tutors as text hints or scaffolds and then quality checked by OATutor editors.Authors could provide any number of hints or scaffolds for a problem step.
4 EXPERIMENT SETUP
Participants complete pre-test, acquisition, and post-test phases after assignment to a hint condition and an algebra lesson. The experiment recruits qualified Mechanical Turk workers and logs their tutoring activity.
- 4.1 Experimental design: Each condition includes a three-item pre-test, five-item acquisition phase, and three-item post-test repeating the pre-test items.Participants are randomly assigned to manual or ChatGPT hints and to Elementary or Intermediate Algebra.
- 4.1 Experimental design: The OATutor system logs condition, lesson, anonymized user ID, problem correctness, hint requests, and timestamps.These records are sent to the researchers’ Firebase account for later analysis.
- 4.2 Participants: Participants are recruited from Mechanical Turk with at least a high school degree and the platform’s master designation.The target is 20 participants for each lesson-condition pairing, or 80 participants overall.
5 RESULTS
Across both algebra lessons, human-authored hints produced significantly greater learning gains than ChatGPT-generated hints, although all conditions showed positive gains.
- Learning gains were higher for human-authored hints than ChatGPT hints in both algebra lessons, with statistically significant differences in each comparison.Both comparisons had p = 0.038.
- The study included 77 participants, and learning gain was calculated as each participant’s average post-test score minus average pre-test score.Mann–Whitney U tests were used after normality was rejected.
- All conditions showed positive learning gains, but only the human-hint conditions had statistically significant pre-to-post improvements.For human hints, p = 0.0219 in Elementary Algebra and p = 0.0213 in Intermediate Algebra; ChatGPT conditions were not significant.
- Intermediate Algebra began with unequal groups: the human-hint control averaged 50.94% at pre-test versus 80.05% for the ChatGPT condition.The groups were even at pre-test in Elementary Algebra but not Intermediate Algebra.
6 CONCLUSIONS AND DISCUSSION
The study concludes that ChatGPT hints can support algebra learning but currently require human oversight, while several design and sampling factors constrain interpretation and motivate further research.
- ChatGPT hints had a 30% quality-based rejection rate, with every rejected hint containing an incorrect answer or incorrect solution steps.No rejected hints contained inappropriate language, poor spelling, or grammatical errors.
- Human hints outperformed ChatGPT hints in both lessons, but the Intermediate Algebra comparison was confounded by the ChatGPT group’s higher pre-test score and near-ceiling performance.The ChatGPT Intermediate Algebra group averaged 80.05% at pre-test versus 50.94% for control.
- The learning-gain comparison may reflect differences between worked solutions and hint scaffolds, not only differences in machine- versus human-authored content.The authors propose comparing manually authored worked solutions or ChatGPT-generated scaffolding in future work.
- Immediate correctness feedback was included in learning gains, so a feedback-only condition would help isolate the effect of hints.Such a condition would tell students whether answers were correct without showing hints.
- The authors recommend larger samples and more advanced material because variable prior knowledge and pre-test scores may produce ceiling effects.The study recruited 77 completers out of 80 and observed substantial Intermediate Algebra pre-test variation.