Source-linked AI summary
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, Majd Sakr
TL;DR
Educators need efficient ways to create current, high-quality programming assessments, but LLM-based generation of programming MCQs has been underexplored. The study develops a GPT-4 system using course context and learning objectives, finding generated MCQs comparable in quality to human-crafted questions and strongly aligned with objectives.
Problem
High-quality programming MCQs require substantial expertise, time, and cost, while LLM use for generating such questions has been underexplored.
Method
The study develops a GPT-4 pipeline that generates MCQs from high-level course context and module-level learning objectives, evaluating quality and alignment.
Results
GPT-4-generated MCQs were comparable in quality to human-crafted questions, with 81.7% passing all evaluation criteria, and showed strong learning-objective alignment.
Takeaways & Limitations
LLM-powered tools could reduce educators’ assessment-authoring workload and make assessment updating and revision more efficient.
Takeaways & Limitations
The study did not evaluate generated MCQs in live classrooms, so their pedagogical impact and effectiveness in assessing student learning remain unknown.
Abstract
from arXiv · showhide
There is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding exercises, the use of LLMs for generating programming MCQs has not been extensively explored. We analyzed the capability of GPT-4 to produce multiple-choice questions (MCQs) aligned with specific learning objectives (LOs) from Python programming classes in higher education. Specifically, we developed an LLM-powered (GPT-4) system for generation of MCQs from high-level course context and module-level LOs. We evaluated 651 LLM-generated and 449 human-crafted MCQs aligned to 246 LOs from 6 Python courses. We found that GPT-4 was capable of producing MCQs with clear language, a single correct choice, and high-quality distractors. We also observed that the generated MCQs appeared to be well-aligned with the LOs. Our findings can be leveraged by educators wishing to take advantage of the state-of-the-art generative models to support MCQ authoring efforts.
1 INTRODUCTION
The paper examines whether GPT-4 can generate high-quality programming MCQs aligned with specified learning objectives. It presents an LLM-based pipeline for Python courses and evaluates generated questions against established quality requirements and LO alignment.
- Motivation: GPT-4 MCQ generation is investigated as a response to the expertise, time, and expense required to create high-quality assessments, particularly for code-containing questions.MCQ authoring in computing education also faces distinct challenges related to including pieces of computer code.
- Approach: The study develops an LLM-based GPT-4 pipeline that uses high-level course context and detailed module-level learning objectives to generate well-formed programming MCQs.The intended outputs use clear language, plausible distractors, and alignment with the specified learning objectives.
- Research questions: The evaluation asks whether generated MCQs provide sufficient information, use clear language, have one correct answer and high-quality distractors, and contain syntactically and logically correct code.These criteria comprise the study’s first research question concerning typical MCQ quality requirements.
- Research questions: A second research question examines how well generated MCQs align with the specified learning objectives.The study treats LO alignment as a distinct evaluation dimension alongside question quality.
- Contributions: The stated contributions include an early study of LLM-based MCQ generation for programming classes and generation from learning objectives rather than only short course-material excerpts.The authors also describe the work as one of the most extensive and detailed evaluations of generated MCQs, including alignment with learning objectives, covering 1,100 MCQs.
2 RELATED WORK
Prior automated MCQ research commonly separates stem, answer, and distractor generation and often relies on textual context. This study positions GPT-4 as a single-step, end-to-end alternative for programming MCQs, an area with limited prior work.
- Element generation: Automated MCQ research commonly targets individual elements, especially stems, keys, or distractors, with question answering often serving as key generation.Prior work spans reading comprehension, cloze tests, and non-computing domains such as biology.
- Distractor generation: Distractor-generation systems typically use the stem or key to produce plausible options, sometimes generating extra candidates and ranking them.Technical domains may require additional resources such as domain-specific ontologies.
- End-to-end generation: Existing end-to-end systems generate stems, keys, and distractors together, but reported evaluations are limited, including ten questions in one system and no evaluation in another.These systems use T5 and treat the component-generation tasks separately from this paper’s GPT-4 approach.
- Computing education: Computing-education MCQ generation has received comparatively little attention, with prior work shifting toward learner-sourcing and only recent GPT-3/GPT-4 studies emerging.LLMs had already demonstrated effectiveness in producing other educational artifacts such as readings, code explanations, solutions, feedback, and help responses.
3 DATA SET
The dataset combines learning objectives from six Python-related higher-education courses with manually collected MCQs used for comparison. Human-crafted questions were manually associated with learning objectives and filtered to retain single-answer items.
- Learning objectives: The dataset contains 246 module-level learning objectives from four Python programming courses and two introductory data science courses.
- Human-crafted questions: Three courses supplied 529 manually collected MCQs for comparison with automatically generated questions.
- Data curation: Researchers manually assigned each collected MCQ to its best-aligned learning objective and excluded 51 questions that could not be assigned reasonably.Questions requiring more than one correct choice were also excluded because the study focused on single-answer MCQs.
4 MCQ GENERATION
The pipeline generates Python-programming MCQs from high-level course and module information by combining predicted Bloom’s levels, question-type resources, and GPT-4 prompting. Its design targets LO alignment and standard MCQ quality requirements through structured resources, examples, and output constraints.
- Pipeline overview: The pipeline supplies course, module, and LO information alongside internal design resources to GPT-4, which outputs the stem, correct answer, and distractors.The user provides high-level course and module context plus a targeted LO; prompts combine this information with curated resources.
- Design resources: Curated design resources provide MCQ principles, Bloom’s taxonomy definitions, question-type examples, course descriptions, and a JSON output format.Examples are selected by question type so generated questions target cognitive processes appropriate to the LO.
- LO and question-type mapping: A BERT classifier predicts the LO’s Bloom’s taxonomy level, which is used to select appropriate MCQ types and included in the user prompt.The classifier was fine-tuned on 21,380 LOs from 5,558 university courses, with predictions covering six Bloom’s levels.
- LO and question-type mapping: The system maps Bloom’s levels to question types such as recall, fill-in-the-blank, scenario-based, and correct-output questions to target distinct cognitive processes.The mapping is intended to avoid complex code-tracing questions for LOs focused on remembering concepts.
- Prompt constraints: The prompting guidance requires clear, concise, grammatically correct questions and distractors that are plausible, related to the stem, and definitively incorrect.The prompt also specifies that each MCQ has one correct answer and two distractors.
5 EXPERIMENTAL DESIGN
The study evaluated GPT-generated and human-crafted Python MCQs using a six-criterion rubric, with multiple annotators and statistical comparison procedures.
- Evaluation dataset and rubric: 651 automatically generated and 449 human-crafted MCQs were evaluated across six rubric criteria covering question quality and LO alignment.The generated questions covered 246 learning objectives, while the rubric’s first five criteria addressed RQ1 and its sixth addressed RQ2.
- Annotation procedure: Each of the 1,100 MCQs received annotations from students and instructors, producing 3,076 annotations overall.Each question was annotated by at least one student and one instructor, averaging slightly fewer than three annotations per MCQ.
- Annotation procedure: Annotators attempted each question before applying the rubric, and disagreements were resolved through majority vote, instructor precedence, correctness, and least-favorable evaluation.The rules were applied in that order.
- Inter-rater agreement: Overall Fleiss κ was 0.22, while Gwet’s AC1 ranged from 0.62 to 0.96 across rubric items.The study used Gwet’s AC1 because κ can underestimate agreement when one label is dominant.
- Statistical analysis: GPT-generated and human-crafted MCQs were compared across all six rubric categories using Fisher’s exact test for statistically significant differences.The test was extended to multiple categories through approximation.
6 RESULTS
GPT-generated MCQs were broadly comparable in quality to human-crafted questions and appeared better aligned with the specified learning objectives, despite weaknesses in answer uniqueness and distractors.
- MCQ Quality (RQ1): 4.9% of GPT-generated MCQs had multiple correct choices versus 1.1% of human-crafted MCQs.This difference was statistically significant for the presence of the correct answer, with p = 0.002.
- MCQ Quality (RQ1): GPT-generated MCQs were more likely than human-crafted ones to contain distractors that revealed the correct answer, at 4.0% versus 0.9%.The difference for obviously wrong options was statistically significant, with p = 0.002.
- MCQ Quality (RQ1): Overall, generated MCQs appeared comparable in quality to human-crafted questions, providing sufficient information in clear language and syntactically and logically correct code.Their main weaknesses concerned having a single correct answer and maintaining high-quality distractors.
- MCQ-LO Alignment (RQ2): GPT-generated MCQs appeared noticeably better aligned with learning objectives than human-crafted MCQs, with Fisher’s exact test yielding p < 10^-9.Human-crafted questions were more often misaligned with the intended cognitive level or unsuitable for the LO, and were unrelated to the LO more often than generated questions.
7 DISCUSSION
GPT-4-generated MCQs generally approached human quality and were usually well-aligned with learning objectives, but some generated questions contained serious validity problems requiring human review.
- 7 DISCUSSION: Generated MCQs were more likely than human-crafted questions to contain multiple correct answers, especially at Bloom’s Apply and Create levels.This issue affected 4.9% of generated MCQs and requires human intervention because GPT-4 sometimes marked only one of several correct choices.
- 7 DISCUSSION: Generated MCQs were more likely than human-crafted questions to include obviously wrong distractors, particularly in fill-in-the-blank and scenario-based questions.Correct answers were also sometimes revealed by code included in the question stem.
- 7 DISCUSSION: 81.7% of generated MCQs passed all evaluation criteria, indicating GPT-4’s overall effectiveness was close to human performance.The authors state that fewer than 1 in 5 generated questions would require instructor edits.
- 7 DISCUSSION: The proposed pipeline generated one MCQ per learning objective, and generated questions were usually better aligned with objectives than human-crafted questions.The authors attribute this advantage partly to educators emphasizing module topics rather than precise learning-objective alignment.
- 7 DISCUSSION: Alignment failures were largely independent of Bloom’s level, MCQ type, learning objective, and course, although Python Essentials 1 showed slightly worse relative alignment.The authors caution that manually pairing human-crafted questions with objectives may explain part of the observed alignment difference.
8 IMPLICATIONS FOR TEACHING PRACTICE
The findings suggest that LLM-powered tools can support teaching practice by producing MCQs comparable in quality to human-authored questions while improving learning-objective alignment.
- 8 IMPLICATIONS FOR TEACHING PRACTICE: LLM-powered tools could reduce instructors’ assessment-authoring workload and make updating and revising assessments more efficient.The authors suggest that this could leave instructors more time for student engagement and curriculum enhancement.
- 8 IMPLICATIONS FOR TEACHING PRACTICE: Automatically generated MCQs appeared better aligned with learning objectives than human-crafted MCQs, suggesting possible value for assessment design and curriculum planning.The authors suggest this alignment could support more accurate representations of students’ mastery of intended objectives, though generated questions still require responsible editing.
9 LIMITATIONS AND THREATS TO VALIDITY
The study’s validity is constrained by its evaluation design, limited rater pool, manual question-objective pairing, and focus on Python courses in higher education.
- 9 LIMITATIONS AND THREATS TO VALIDITY: Potential validity threats include bias from the limited human-rater pool and evaluating pre-existing human questions with a rubric their authors had not seen.These conditions may make the comparison between generated and human-crafted questions imperfect.
- 9 LIMITATIONS AND THREATS TO VALIDITY: The study did not evaluate generated MCQs in live classrooms, so their pedagogical impact and effectiveness for assessing student learning remain unknown.The study also did not compare difficulty with human-crafted questions or alignment with learning content.
- 9 LIMITATIONS AND THREATS TO VALIDITY: Manual pairing of human-crafted MCQs with learning objectives may have contributed to the observed human-question misalignment.This introduces uncertainty into the comparison of alignment between generated and human-crafted questions.
- 9 LIMITATIONS AND THREATS TO VALIDITY: Because the study covered only higher-education Python courses, its findings may not generalize to other programming languages or education levels.The authors note that LLM performance may differ in other languages or specialized domains.
10 CONCLUSIONS AND FUTURE WORK
The study provides promising evidence that GPT-4 can generate high-quality, learning-objective-aligned MCQs for Python programming courses, while identifying several directions for further validation and extension.
- 10 CONCLUSIONS AND FUTURE WORK: LLM-powered systems generated Python programming MCQs comparable in quality to human-authored questions and strongly aligned with learning objectives.The authors present this as evidence of the feasibility of high-quality automated MCQ generation.
- 10 CONCLUSIONS AND FUTURE WORK: Future work should test generated MCQs in real classrooms and examine their uptake, utility, discriminative power, novelty, and diversity.The authors also call for research on generating complete quizzes and quiz pools for particular learning objectives.
- 10 CONCLUSIONS AND FUTURE WORK: Further studies should evaluate refinement through feedback, parameter settings such as temperature, and generalization to other programming languages.The current study did not assess GPT-4 refinement capabilities or the effects of different parameter settings.