Source-linked AI summary
Large Language Models in Computer Science Education: A Systematic Literature Review
Nishat Raihan, Mohammed Latif Siddiq, Joanna C. S. Santos, Marcos Zampieri
TL;DR
Evidence on LLMs’ impact in CS education remains incomplete, and the review synthesizes research across educational settings and uses. Across the reviewed studies, work is concentrated in undergraduate and introductory education, with students generally positive but instructors concerned about learning effectiveness and potential over-reliance.
Problem
The review addresses the lack of a comprehensive synthesis of LLMs’ impact in CS education, an area where students already use these tools.
Source: §1 · pp. 1–2Method
Results
Most reviewed research concerns undergraduate and introductory programming courses, while students are usually positive and instructors worry about learning effectiveness and potential over-reliance.
Source: §5 · p. 5Takeaways & Limitations
The review concludes that educators are gradually adopting LLMs in courses, while most CS curricula still need changes to accommodate recent AI advances.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Large language models (LLMs) are becoming increasingly better at a wide range of Natural Language Processing tasks (NLP), such as text generation and understanding. Recently, these models have extended their capabilities to coding tasks, bridging the gap between natural languages (NL) and programming languages (PL). Foundational models such as the Generative Pre-trained Transformer (GPT) and LLaMA series have set strong baseline performances in various NL and PL tasks. Additionally, several models have been fine-tuned specifically for code generation, showing significant improvements in code-related applications. Both foundational and fine-tuned models are increasingly used in education, helping students write, debug, and understand code. We present a comprehensive systematic literature review to examine the impact of LLMs in computer science and computer engineering education. We analyze their effectiveness in enhancing the learning experience, supporting personalized education, and aiding educators in curriculum development. We address five research questions to uncover insights into how LLMs contribute to educational outcomes, identify challenges, and suggest directions for future research.
1 Introduction
LLMs offer CS students coding help but can also enable assignment completion and raise concerns about misuse, learning, and academic integrity. Because their impact remains incompletely understood, this review examines the evidence to inform educational practice.
1 Introduction
- Existing plagiarism-detection methods often perform unreliably on output from recently introduced LLMs, while students may use these models to complete beginner-course assignments.Researchers have also examined model performance on datasets and past assignments, as well as identifying AI-generated work.Source: §1 · p. 1
- LLMs may support learning with quick coding feedback and explanatory code examples, and CS courses can prepare students for workplaces where these tools are common.The paper conducts a systematic literature review using five research questions and reports that its scripts and data are available in a GitHub repository.Source: §1 · pp. 1–2
- The review addresses a gap in comprehensive evidence on LLMs’ impact in CS education, where understanding benefits and challenges matters for curriculum design and assessment.Prior surveys covered programming exercise generation, security and privacy, software development, software engineering education, or programming alone.Source: §1 · pp. 1–2
2 Methodology
The authors conduct a systematic review to characterize LLM use in CS education across five research questions. They search multiple databases, screen studies using stated criteria, and extract and review information relevant to those questions.
2 Methodology
- The review follows three stages: planning the review, conducting it, and reporting its results.The authors set five research questions, select studies using specified criteria, and organize findings around those questions.Source: §2 · p. 2
2.1 Research Questions
- The five research questions cover educational levels, CS sub-disciplines, research methodologies, programming languages, and LLMs used.The questions aim to identify educational stages, target areas and practices, and patterns in language and model selection.Source: §2.1 · p. 2
2.2 Search Method
- A search query combining CS or software-related terms with education and LLM terms was applied across eight library databases and returned 1,735 papers.The databases include ACM Digital Library, IEEE Xplore, Scopus, ACL Anthology, Web of Science, Springer Link, Science@Direct, and ArXiv.Source: §2.2 · p. 2
2.3 Inclusion and Exclusion Criteria
- Screening reduced 1,735 initial studies to 1,423 after deduplication and date limits, then to 187 after title and abstract review, and finally to 125 included papers.The authors applied their inclusion and exclusion criteria at the screening stages.Source: §2.3 · p. 2, Table 1 · p. 2
2.4 Data Extraction
- The authors extracted educational level, CS discipline, programming languages, LLMs, and research methods from the selected studies; a second author peer-reviewed the extraction.Source: §2.4 · p. 2
3 Results
Across 125 reviewed papers, research concentrates on undergraduate and introductory CS education, with Python prominent and case-based studies common. The review also finds varied model use and comparatively limited attention to advanced topics and benchmarking.
3.1 RQ1: Educational Levels
- 111 reviewed papers focus on undergraduate CS courses, compared with 15 on graduate courses, 4 on PhD courses, and 2 on K–12 education.The review also identified one study of ChatGPT for employee training in a software-engineering workplace; higher-level content limitations and practical issues including GPT-4 cost are cited as factors in the undergraduate focus.Source: Figure 1 · p. 3, §3.1 · p. 3
3.2 RQ2: CS Sub-disciplines
- 67% of works focus on introduction to programming and introduction to CS; advanced CS concepts receive limited focus, suggesting further exploration of LLMs’ ability to help teach them.Disciplines represented by fewer than 3 papers were aggregated as “Others” in Table 2, including Distributed Systems, Operating Systems, Software Testing, and Databases.Source: §3.2 · p. 3
3.3 RQ3: Research Methodologies
- Case studies and ethnography appear in 38% of papers, action research in 24%, and benchmarking in 5%.One study used semi-structured interviews with LLMs about the relevance of computer science in K–12 education.Source: §3.3 · pp. 3–4, Table 3 · p. 4
3.4 RQ4: Programming Languages
- Python is the most studied programming language at 55% of papers, while Java is second and JavaScript, C, and C++ receive less attention.The review attributes Python’s prominence in part to introductory CS courses; TypeScript and R tie for tenth place among the top ten languages.Source: Figure 2 · p. 4, §3.4 · p. 4
- Reviewed LLM uses include code generation, code-related Q&A, multiple-choice questions, student and teacher perceptions, course feedback, and scaffolding.The review therefore covers tasks and broader course-related uses, not code generation alone.Source: §3.4 · p. 4
3.5 RQ5: Most commonly used LLMs
- Most reviewed papers use ChatGPT, primarily its older GPT-3.5 version; the review links this pattern to study timing and GPT-4’s higher cost.Twenty-six models appeared only once across 12 papers; a work could use more than one model.Source: Figure 3 · p. 4, §3.5 · p. 4
4 Discussion
The discussion synthesizes student and instructor sentiment, task completion, adoption patterns, and expectations for future use of LLMs in CS education.
Students’ and instructors’ sentiment about using LLMs in CS Education
- Students generally report positive experiences and value LLM-generated examples and explanations, while instructors worry about over-reliance, plagiarism, and independent problem-solving.Students also report frustration with prompting and finding accurate responses [84].Source: §4 · pp. 4–5
Task completion with LLMs and LLM-based tools
- Task completion is mixed: LLMs can help with introductory problems, buggy-code repair, and code quality, but struggle with machine-learning and some non-English tasks [80] [62] [39].LLMs are generally better at writing code than answering questions, and can generate problems and feedback [101] [89].Source: §4 · p. 5
Adoption of LLMs in different use cases
Expectation and future direction of using LLMs in CS Education
5 Conclusion
The review synthesizes 125 relevant studies to map research on LLMs in CS education. Its findings show a concentration in undergraduate and introductory programming contexts, alongside concerns about learning effectiveness and a need for curricular change.
5 Conclusion
- The review screened 1,735 papers to 125 relevant studies and addressed five RQs spanning educational levels, sub-disciplines, methodologies, and programming languages.It also discusses LLM adoption, student sentiment, and future directions.Source: §5 · p. 5
- The evidence base centers on undergraduate and introductory programming, case-based studies, and Python; students are usually positive while instructors worry about learning effectiveness and over-reliance.The review reports that most current research focuses on undergraduate education and introductory programming courses.Source: §5 · p. 5
- Educators are gradually adopting LLMs, but most CS curricula still need changes to accommodate AI advances and inform future research and applications.Source: §5 · p. 5