Source-linked AI summary
Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
Wenhan Lyu, Yimeng Wang, Tingting, Chung, Yifan Sun, Yixuan Zhang
TL;DR
Introductory computer science education lacks sufficient long-term, real-world evidence about the effects of LLM-powered tools on students. This paper conducts a semester-long between-subjects study with 50 students using CodeTutor and finds improved final scores, while attitudes toward the tool and human teaching assistants change over time.
Problem
Few empirical studies examine the long-term effects of LLM-powered tools on students in introductory computer science education.
Method
The study uses a semester-long between-subjects field study with 50 students and the LLM-powered CodeTutor assistant.
Results
Students using CodeTutor showed significant final-score improvements over controls, with first-time LLM-tool users experiencing the largest gains.
Takeaways & Limitations
CodeTutor supported comprehension, syntax learning, coding tasks, and debugging, but students remained concerned about critical-thinking support and increasingly preferred human assistants.
Takeaways & Limitations
The small study scale limits generalizability, and findings may not translate directly to intermediate or advanced programming courses.
Abstract
from arXiv · showhide
The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.
1 INTRODUCTION
LLMs create both opportunities and risks for introductory computer science education, but their longer-term effects and responsible integration remain insufficiently understood. This study addresses that gap through a semester-long field study of CodeTutor.
- LLMs can complete many assignments in introductory programming courses, raising concerns about misuse, educational quality, and academic integrity.
- When used ethically, LLMs can provide instant coding feedback, generate illustrative examples, and familiarize students with technologies increasingly used in production.
- The impact of LLMs on computer science education remains unknown because their use is difficult to detect and traditional plagiarism detection is insufficient.
- The study asks whether LLM tools affect learning outcomes, how student attitudes change, and how students engage with these tools.
- A longitudinal, between-subject field study followed 50 students from September to December 2023 using the researchers’ web-based CodeTutor tool.
- The work contributes empirical evidence on learning outcomes, engagement patterns, feedback loops, and design implications for LLM-powered programming education.
2 RELATED WORK
Prior intelligent tutoring and LLM research shows educational promise but leaves important questions about scope, cost, long-term effects, and individual differences unresolved. This study addresses these gaps through an extended field evaluation of LLM-powered programming support.
- Intelligent tutoring systems personalize instruction but remain complex and costly to build, domain-limited, and difficult for nonexperts to develop.
- LLM tools have broad accessibility and can support diverse tasks, increasing their adoption in educational settings.
- Few empirical studies examine the emerging impacts of LLMs in education, creating a need for research on long-term effects and potential negative consequences.
- Earlier CodeHelp research linked tool usage with course performance but focused on debugging and did not compare outcomes with traditional teaching assistants.
- Prior work indicates that individual differences influence intelligent tutoring effectiveness, while their effects on LLM-tool interactions and perceptions remain sparsely studied.
- This work evaluates LLM-powered programming support over an extended period to examine practical use and longer-term implications for learning and teaching.
3 METHOD
The study uses a browser-based CodeTutor application and a field evaluation with 50 entry-level computer science students. The method combines system development, participant characterization, and longitudinal data collection.
- The study section covers CodeTutor’s design, participants, procedures, data collection, and quantitative and qualitative analyses.
- CodeTutor is a TypeScript-based browser application integrating OpenAI’s GPT-3.5 model through an API for interactive programming support.
- The IRB-approved field study involved 50 participants from a United States four-year university’s computer science department.
- Participants were entry-level undergraduate computer science students aged 18 or older who could speak and write English.
3.3 Study Procedure & Data Collection
The semester-long procedure established a pre-test baseline, assigned students to control or CodeTutor groups, and collected feedback and academic outcomes during naturalistic course participation. Final grades combined multiple assessment components with differing access to CodeTutor.
- The field study ran from September 27 through December 11, 2023, covering the course period through the final exam.
- 3.3.1 Pre-test: Students completed a 22-question Python pre-test spanning three difficulty levels to establish baseline programming knowledge.
- 3.3.1 Pre-test: CodeTutor’s interface included conversation history, a main dialogue area, conversation-level feedback, and message-level feedback.
- Participants reported prior LLM experience and demographics, including major, gender, and race or ethnicity, before analysis.
- 3.3.2 Control vs. Experimental Group: The control group used traditional learning methods with human TAs, while the experimental group used CodeTutor as its primary out-of-class educational tool.
- 3.3.2 Control vs. Experimental Group: Students were stratified by prior LLM use and pre-test performance before assignment, while CodeTutor use remained self-paced and other LLM tools were prohibited.
- Final grades averaged labs, assignments, midterms, and the final exam; CodeTutor was allowed for labs and assignments but not exams.
3.4 Data Analysis
The study analyzed CodeTutor use through descriptive, inferential, correlational, regression, and thematic methods covering scores, attitudes, prompts, and interactions.
- Descriptive statistics summarized categorical frequencies and percentages and continuous means and standard deviations.
- Paired t-tests compared pre-test and post-intervention scores within experimental and control groups.
- Multiple regression with family-wise p-value adjustment examined score improvement while accounting for prior LLM experience and demographics.
- ANOVA assessed CodeTutor accessibility and academic performance, while chi-squared testing examined associations between prompt quality and CodeTutor performance.
- Spearman correlations, Kruskal-Wallis testing with post-hoc comparisons, and thematic analysis examined attitudes and conversational interactions.
4 RESULTS
The results section evaluates CodeTutor’s effects on academic performance, student attitudes, and engagement patterns in entry-level programming courses.
- The analysis addresses learning outcomes, attitudes toward CodeTutor, and engagement patterns across three results subsections.
4.1 RQ1: Learning Outcomes with CodeTutor
CodeTutor users improved more than controls, including on assessments where the tool was unavailable, with larger gains among students new to LLM-powered tools.
- 12.50 points was the experimental group’s average pre-test-to-final-score increase, compared with a 3.17-point average decrease for controls.
- The experimental group’s pre-test-to-final-score improvement was statistically significant (t = -2.847, p = 0.009), unlike the control group’s change (t = -0.879, p = 0.394).
- 7.33 points was the experimental group’s average improvement on CodeTutor-Not-Allowed components, with a significant pre-test-to-final-exam gain (t = -2.405, p = 0.026).
- 102.29 was the CodeTutor-Allowed mean score versus 93.40 for CodeTutor-Not-Allowed components, a significant difference (t = 2.31, p = 0.03).
- Students who had never used LLM-powered tools previously had higher experimental-group final scores than prior users (β = 18.877, p = 0.032).
- Data science, mathematics, and biology majors showed higher final scores than computer science majors, while gender and race differences were not significant.
4.2 RQ2: Students’ Attitudes towards CodeTutor
Students valued CodeTutor’s comprehension and syntax support more than its critical-thinking role, while confidence in its comprehension and preference over human TAs declined over time.
- Descriptive Analysis: 67% agreed or strongly agreed that CodeTutor understood their intended questions, whereas 35% disagreed or strongly disagreed that it helped them think critically.
- Descriptive Analysis: Comprehension and Syntax Mastery were moderately positively correlated, while Syntax Mastery and Independent Learning showed a strong correlation.
- Relationships in Student Attitudes: Students’ attitudes differed across questions (χ2 = 32.99, p < 0.05), with less agreement about critical-thinking support than comprehension, syntax learning, or TA replacement.
- Relationships in Student Attitudes: Agreement with CodeTutor’s comprehension decreased over time (β = -0.114, p < 0.01), and TA Replacement also declined weakly (β = -0.099, p < 0.1).
- Relationships in Student Attitudes: Students recognized CodeTutor’s query comprehension and syntax assistance but questioned its capacity to promote critical thinking and increasingly preferred human teaching assistants.
4.3 RQ3: Students’ Engagement with CodeTutor
The study characterized how students engaged with CodeTutor through diverse programming inquiries, feedback exchanges, and prompt-quality analysis. Clearer, more detailed prompts were significantly associated with more effective responses.
- Engagement volume: 82 conversation sessions contained 2,567 messages, covering 415 unique topics.Sessions averaged 5.06 topics and 6.19 messages per topic.
- Interaction volume: 1,288 messages came from users and 1,279 from CodeTutor, indicating a nearly balanced exchange.
- Interaction types: User interactions included programming-task inquiries, syntax questions, general questions, greetings, modification requests, debugging, and follow-up feedback.CodeTutor responses were categorized as corrections or explanations.
- Prompt quality: 37% of 1,190 non-greeting prompts were classified as good quality, while 63% were classified as poor quality.Good prompts provided sufficient detail for accurate responses; poor prompts did not.
- Prompt effectiveness: χ^2 = 144.84, p < 0.001, showing a significant correlation between prompt quality and response effectiveness.Clearer and more detailed prompts were associated with responses more likely to be effective.
5 DISCUSSION
The discussion interprets CodeTutor’s educational benefits and limitations across learning outcomes, student engagement, prompt quality, and use over time. It emphasizes Generative AI literacy, critical evaluation, and balancing LLM support with human instruction.
- Educational value: Students valued CodeTutor for comprehension, syntax learning, coding tasks, and debugging, but questioned its ability to cultivate critical thinking.
- Generative AI literacy: 63% of student-generated prompts were unsatisfactory, indicating a need to develop students’ Generative AI literacy.The paper defines this literacy as effectively interacting with AI tools and formulating and interpreting queries and responses.
- Design implications: The authors recommend integrating Generative AI literacy, critical-thinking exercises, prompt templates, and complementary human instruction into programming education.
- Temporal dynamics: Students’ confidence in CodeTutor’s accuracy and preference for it as a TA replacement declined with prolonged use, while attitudes toward syntax mastery and independent learning remained stable.
- Completion and comprehension: Assignments and labs permitting CodeTutor produced higher scores, whereas exams prohibiting it were used to reflect students’ internal programming knowledge.The authors caution that permitted tasks may prioritize completion over deep comprehension.
- Student practices: Nearly half of low-quality prompts classified as over-reliance copied original assignment questions, suggesting quick-fix use rather than engagement with question logic.
6 LIMITATIONS AND FUTURE WORK
The study’s findings are constrained by its small, introductory-course sample, reliance on GPT-3.5, and difficulty controlling the semester-long experimental environment.
- Scope: The study involved a relatively small sample, limiting the generalizability of its findings.Future work will test larger and more diverse student populations and settings.
- Course level: Because the study focused on beginning-level computer science courses, its findings may not directly apply to intermediate or advanced programming courses.
- Model: Reliance on GPT-3.5 may have affected tutoring quality because its responses were not always accurate or contextually appropriate.
- Study design: Controlling the experimental environment, particularly the control group, was challenging in the semester-long study.The authors call for more experimental designs to clarify factors affecting student learning.
7 CONCLUSION
In a semester-long study of 50 students, CodeTutor improved final scores, especially for first-time LLM-tool users, while students recognized its practical utility but increasingly preferred human support and questioned its critical-thinking benefits.
- Study design: 50 students participated in a semester-long between-subjects study of CodeTutor in introductory programming.
- Learning outcomes: Students using CodeTutor showed significant final-score improvements over the control group, with first-time LLM-tool users experiencing the largest gains.
- Student perceptions: Students valued CodeTutor for query understanding, syntax learning, programming tasks, and debugging but raised concerns about critical-thinking development.
- Support preferences: Students increasingly preferred human assistant support over CodeTutor despite its usefulness for completing programming tasks.
- Implications: Detailed and clear prompts produced more accurate responses, supporting the integration of Generative AI literacy and critical-thinking instruction into curricula.