PaperlayerView exact revision on arXiv

Source-linked AI summary

The Robots are Here: Navigating the Generative AI Revolution in Computing Education

James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Peterson, Raymond Pettit, Brent N. Reeves, Jaromir Savelka

arXiv:2310.00658v1cs.CYcs.AIcs.HC

TL;DR

Generative AI is changing computing education and raising questions about pedagogy, assessment, policy, and ethics. This report synthesizes literature, surveys stakeholders, examines instructional adaptations, discusses ethics, and evaluates LLM performance, finding strong capabilities alongside rapidly changing evidence and practice.

  • Problem

    Students now actively use generative AI tools in computing education, intensifying longstanding concerns about easy ways out, intellectual laziness, and illusions of achievement at scale.

  • Method

    The report combines a literature review, stakeholder surveys, educator perspectives, ethical analysis, and benchmarking of LLMs on computing-education datasets.

  • Results

    Current models tend to perform at least as well as most students on typical introductory-level programming tasks, while students and instructors generally report similar perceptions of LLMs.

  • Takeaways & Limitations

    Educators should acknowledge LLMs, clarify policies, assume use, reconsider learning objectives, and recognize that generated solutions may resemble student work.

  • Takeaways & Limitations

    The review covered literature published from 2021 to August 2023, used one snowballing step, and included some non-peer-reviewed sources.

Abstract

from arXiv · show

Recent advancements in artificial intelligence (AI) are fundamentally reshaping computing, with large language models (LLMs) now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.

1 INTRODUCTION

Generative AI is changing computing education by enabling students to actively use LLMs for code and natural-language tasks, while challenging established teaching and assessment practices. These tools also create opportunities for supported programming assistance and personalised educational resources.

  • ChatGPT’s public release accelerated students’ active use of generative AI as a tool for producing new data through natural-language prompts.
  • LLMs increasingly generate and interpret source code, enabling possible student–LLM pair programming and assignment assistance.Many models are freely available to students, and early reports indicate existing assignment use.
  • Introductory programming exercises and automated assessment may need reconsideration because LLMs can solve many such problems with simple prompts.
  • LLM use raises concerns about incorrect or biased answers, over-reliance, plagiarism, incomprehensible generated code, and ineffective AI detection.
  • With appropriate context, scaffolding, and support, LLMs can act as programming partners, explain code, clarify errors, and help instructors generate personalised teaching materials.
  • The report frames LLMs as likely to affect what, how, and whom educators teach, while documenting early developments and effective practices.

2 CONTRIBUTIONS

The report combines literature review, stakeholder surveys, educator interviews, ethical analysis, and replication-oriented work to describe LLM issues in computing education. Together, these efforts document current practices and establish resources for evaluating and responding to rapid technological change.

  • Reviewing Literature: The report reviews literature on LLMs in computing education and synthesises opportunities and challenges in the domain.
  • Evaluating Current Attitudes: An international survey captures students’ and instructors’ perspectives on LLM experiences, expectations, beliefs, and uses.
  • Identifying New Instructional Approaches: Interviews with instructors examine teaching about and using LLMs in classrooms, including perceived advantages and disadvantages.
  • Exploring Ethical Implications of LLMs: An evidence-based ethical analysis evaluates university AI policies against the ACM Code of Ethics and addresses academic integrity.
  • Encouraging Replication: The report replicates prior work with newer LLMs, identifies openly available datasets, and highlights challenges in research methods and dataset quality.

3 REVIEW OF LITERATURE

The review synthesizes 71 papers on LLMs in computing education, finding broad evidence of substantial impact alongside uneven capabilities, opportunities, and risks. It covers LLM performance, user interactions, student-work analysis, teaching-material generation, educator workload, learning experiences, and assessment redesign.

  • Assessing performance, capabilities and limitations: LLMs generally match or exceed average students on code generation, but perform less well on multiple-choice, Parsons, and computational-thinking tasks.
  • Interactions between programmers and LLMs: Students use AI code generators through interaction modes including exploration, acceleration, and shepherding.
  • Interactions between programmers and LLMs: AI code generators improved novice students’ progress, errors, correctness, and time spent without negative effects on learning.The reported changes were 1.15x progress, 0.59x errors, 1.8x higher correctness, and 0.57x time spent.
  • The reviewed literature broadly agreed that LLMs will substantially affect computing teaching and learning, while performance and support quality remain task-dependent.
  • Opportunities for computing education: LLMs offered educators opportunities to reduce workload, personalize assistance, generate learning resources, and rethink assessment toward code understanding and review.Reviewed papers also identified pair programming, renewed focus on problem solving, and generated exercises with prescribed themes and programming concepts.

4 SURVEY OF STUDENT AND INSTRUCTOR PERCEPTIONS ABOUT GENAI

Students and instructors reported similar GenAI experiences and expectations, while differing or uncertain views emerged around institutional restrictions, ethical use, and classroom adoption. Instructors described limited current uptake but a broad range of educational uses, alongside observations of both student misconduct and productive assistance.

  • Experience and usage: Students and instructors primarily used GenAI for writing code and working with text, with students slightly more experienced in code generation.
  • Course and institutional policies: Both groups supported some coursework restrictions, but students and instructors differed over which restrictions should apply and whether university policies were clear.
  • Expectations and beliefs: Students and instructors agreed that GenAI cannot replace human instructors, yet expected its role to increase in teaching, learning, and careers.
  • Help-seeking preferences: Students still preferred web searches for help, although GenAI surpassed online discussion forums as a preferred assistance source.
  • Ethical use: Instructors generally viewed auto-generating complete assignments without understanding the code as unethical, but disagreed about cases involving comprehension.
  • Instructor use: Most instructors were not currently using GenAI, while users applied it to assessment questions, exercises, examples, writing, research, and summarisation with varying satisfaction.
  • Instructor observations: Fewer than one-third of instructors said they had not observed student GenAI use, while 20% described academic misconduct involving complete generated solutions.

5 CURRICULUM AND ASSESSMENT

Interviewed educators are responding to LLMs in varied ways: some preserve foundational programming objectives while adding competencies such as prompt engineering, problem decomposition, code comprehension, and critical thinking. Others are integrating LLMs as aids while maintaining hands-on coding, and anticipate further changes to curricula and courses.

  • New and evolving content: Educators are also using LLMs to introduce new content, including cybersecurity applications and the role of LLMs in code generation.Examples include new cybersecurity sessions and updating a CS1 course of approximately 50 students.
  • Changing learning objectives: Educators are adding LLM-related objectives such as prompt engineering, problem decomposition, understanding non-determinism, and critically evaluating outputs.These changes are often being made at a fine-grained course level rather than reflected in official syllabi.
  • Future curriculum change: The report suggests LLMs may eventually produce entirely new courses, syllabi, and learning objectives, while educators remain flexible as the technology evolves.One cited example is DeepLearning.AI’s online “Generative AI” course.
  • Preserving foundational skills: Code comprehension, algorithmic thinking, debugging, and communication remain important competencies because students need foundational skills to use LLMs effectively.Interviewees particularly emphasized assessing students’ ability to read, trace, and understand code.
  • Integrating LLMs with coding: Some educators integrate LLMs into programming courses while retaining hands-on coding and practice of the foundational work that LLMs can perform.Others recommend introducing LLMs after students master traditional programming concepts, rather than replacing basic instruction.
  • Program-level alignment: Overall, educators seek constructive alignment between changing activities and assessments and broader program-level outcomes, while balancing LLM-enabled problem solving with core competencies.The discussion presents LLMs as complementary tools rather than substitutes for foundational programming education.

6 ETHICS

The report frames LLM ethics in computing education around societal effects, professional responsibilities, institutional policy, and academic integrity. It highlights risks involving reliability, bias, privacy, confidentiality, unequal access, and harms to student learning and well-being.

  • Ethical analysis should examine training-data provenance, system use, and associated costs rather than only integrating ethics into AI systems.
  • The literature commonly values performance, generalisation, efficiency, novelty, and scalability, but benchmark-based performance can encode bias.
  • The report questions the assumption that LLM access is free because costs may be hidden and students may receive unequal access to more powerful models.
  • Avoid harm: The report links inappropriate LLM use to over-reliance, academic-integrity breaches, incorrect products, and lost learning that can impair long-term subject mastery.
  • Be honest and trustworthy: University policies warn that LLMs can produce incorrect or fabricated information, including hallucinated data, sources, and facts.
  • Be fair and take action not to discriminate: Policies identify discrimination risks because models can reproduce stereotypes and misinformation from their training data.
  • Respect privacy and honor confidentiality: LLM use raises privacy and confidentiality concerns because tools may retain user data, process student records, or expose intellectual property.

7 ACADEMIC INTEGRITY IMPLICATIONS

The report distinguishes several academic-integrity categories for generative-AI use and argues that responsibility depends on authorship, permitted use, and student accountability. It recommends explicit rules, disclosure, and instruction about ethical use.

  • The review distinguishes plagiarism, collusion, contract cheating, falsification, and unauthorised resources as academic-integrity concerns.
  • 7.1 Plagiarism: The report treats the student user as the author of AI-generated work and therefore does not classify permitted AI use itself as plagiarism.
  • 7.1 Plagiarism: AI detectors are unreliable, produce many false positives, and perform even worse when evaluating code.
  • 7.2 Collusion: Because generative AI lacks agency, the report does not classify submitting generated content as collusion.
  • 7.3 Contract cheating: The report rejects treating generative-AI use as contract cheating because it considers the user the author and AI a tool analogous to calculators or other software.
  • 7.4 Falsification: Students remain responsible for inaccuracies, nonexistent citations, and other hallucinations in work produced with generative AI, making falsification relevant.
  • 7.5 Use of unauthorised resources: The report says AI use may be undesirable in introductory programming but appropriate as a productivity tool in some upper-division courses.
  • Educators should state permitted tools explicitly, teach ethical use, and require students to disclose how generative AI was used in assessments.

8 BENCHMARKING LARGE LANGUAGE MODELS FOR COMPUTING EDUCATION

The report benchmarks LLMs on computing-education tasks while examining dataset coverage and evaluation quality. Results show strong performance on many programming problems, but outcomes depend on model, task difficulty, problem clarity, and dataset limitations.

  • 8.1 Review of empirical datasets: The benchmark reviews available datasets, finding that most emphasize introductory Python code-generation exercises rather than student–LLM interactions, code repair, other languages, or advanced coursework.Public datasets also often provide limited evaluation, with incomplete tests sometimes accepting incorrect solutions.
  • 8.3 Replication: The robots are coming: GPT-4 outperformed Codex, solving all but one problem within ten attempts, while GPT-3.5 performed only slightly worse and Copilot solved 20 of 23 exam problems and four of seven Rainfall variants.Failures commonly involved vague descriptions, formatting requirements, negative values, empty-list edge cases, or incomplete completions.
  • 8.4.1 APPS.: Across APPS, GPT-4 averaged 51.5% of test cases versus 39.2% for GPT-3.5, with all-tests pass rates of 36.1% and 21.0%, respectively.GPT-4 scored 72.2% on introductory problems but 28.7% on competition-level problems.
  • 8.4.1 APPS.: The APPS results are constrained because its public online-coding problems may have appeared in recent models’ training data, and some introductory problems are unsuitable for novice programmers.The dataset includes complex competition-oriented tasks even within its introductory set.

9 CONCLUSIONS

This report examines how generative AI is reshaping computing education through evidence synthesis, surveys, interviews, policy analysis, and benchmarking. It presents both significant challenges and opportunities while calling for adaptation, replication, and collective development of practice.

  • Reviewing Literature: The report synthesises findings from 71 primary articles on LLMs in computing education.The review included preprints and assessed articles using quality metrics.
  • Prevailing attitudes: A survey of 171 students and 57 instructors across 20 countries found broadly similar perceptions but differing views about policy clarity.Instructors found course policies less clear than students, while both groups generally supported some usage restrictions.
  • New instructional approaches: Interviews with 22 instructors on five continents documented concrete changes to curricula and assessments.Some instructors were beginning to emphasise process over product.
  • Academic integrity: Policies & recommendations: The report finds that university policies address many ACM ethical principles, but how students learn ethical generative-AI use remains unclear.It recommends explicit course restrictions, teaching ethical use throughout courses, and disclosure of tools used for graded work.
  • Encouraging replication: A replication using a state-of-the-art model showed extraordinary performance improvement over two years, while the report encourages reproducible evaluation.The authors prepared and released problem descriptions and test cases to facilitate future replication.
  • 9 CONCLUSIONS: The report frames LLMs as both significant challenges and new opportunities, urging educators to adapt teaching methods, policies, and shared practice.It presents the work as a snapshot of an early, rapidly evolving stage and as a call to action.

A PAPER EXTRACTION FORM

The paper extraction form standardises how included studies are characterised, evaluated, and connected to computing education. It records bibliographic, methodological, participant, tool, quality, contribution, curriculum, opportunity, threat, and supplementary information.

  • Bibliographic and study information: The form records each paper’s title, BibTeX entry, article type, and author affiliation.Article types include position papers, supervised or unsupervised studies, new tool papers, and evaluation papers.
  • Participants: It captures participant countries, education levels, participant counts, and participant descriptions when human data are collected.The form distinguishes tertiary, school, informal, professional, and non-applicable contexts.
  • Study design: Reviewers record the authors’ motivation, the LLM or tool used, research questions, programming languages, and evaluation approach.Evaluation approaches are coded as qualitative, quantitative, or not applicable.
  • Quality and contribution: A quality assessment checks whether research questions, processes, results, and threats to validity or limitations are clearly addressed.The form also requests a short summary of the article’s contribution and key results.
  • Educational implications: The form asks reviewers to identify potential curriculum changes, opportunities, threats, and additional relevant notes.These fields may include commentary prompted by the paper, even when not explicitly stated by its authors.

B STUDENT SURVEY QUESTIONS

The student survey measures students’ backgrounds, GenAI use, verification practices, beliefs, policy perceptions, ethical judgments, and anticipated educational or employment effects.

  • Participant background: Student demographics include gender, country, study level, degree major, and completed courses with programming components.Listed majors include computer science, software engineering, information technology, computer engineering, and bioinformatics.
  • Current use and verification: Students report how regularly they use GenAI for text, code, and images, and what they do after generating code.Verification options range from immediate use and brief skimming to careful skeptical reading with testing.
  • Learning practices and beliefs: The survey asks students to rank GenAI, forums, online searches, friends, and instructors as sources of help when studying or solving problems.It also measures expected future use, perceived learning harm, guidance quality, and possible teacher replacement.
  • Policies: Students evaluate policy clarity and whether GenAI should be always allowed, conditionally allowed, or always disallowed in programming assignments.Open responses ask them to explain when use should be allowed or disallowed.
  • Ethics and anticipated effects: Students are asked about unauthorized use, ethical boundaries, reliance, trust in generated code, detection, and instructors’ checking practices.Additional questions cover text and code uses, employment prospects, and coursework or exam policies.

C INSTRUCTOR SURVEY QUESTIONS

The instructor survey examines educators’ backgrounds, GenAI practices, beliefs, policy judgments, observed student use, teaching and assessment changes, curriculum needs, and institutional policies.

  • Instructor background: Instructor demographics include gender, country, teaching experience, class sizes, and department or school type.The survey records class sizes from 1–10 students through more than 500.
  • Current use and verification: Instructors report GenAI use for text, code, and images, plus how they verify generated code after production.Verification choices include immediate use, brief skimming, careful skeptical reading, and testing.
  • Beliefs and planned changes: The survey measures instructors’ expectations about future use, learning effects, guidance quality, teacher replacement, and teaching GenAI skills.It also asks whether instructors plan to change assessment practices or curricula.
  • Policies and ethics: Instructors evaluate institutional and course policy clarity and whether GenAI should be always allowed, conditionally allowed, or always disallowed.They also report perceived unauthorized student use and ethical restrictions.
  • Practice and institutional context: Open questions document instructors’ uses of GenAI, teaching and assessment changes, curriculum additions, policy documents, and observed student use.The survey also asks whether instructors would participate in interviews.

D STUDENT GUIDE

Generative AI generates information from prompts and can support learning, but its use requires clear boundaries between productive learning assistance and restricted assessments.

  • Student guide: Generative AI generates information in response to prompts using training data and predictive models.Examples include text-based systems such as ChatGPT and Bard.
  • Student guide: Generative AI tools can increase productivity and support learning, but they can also provide answers without helping students learn.
  • Policy on generative AI: Students may use AI tools during labs and assignments, but not in secure assessments such as tests and exams.

Examples of productive use

Productive uses of generative AI focus on understanding course material, programming constructs, and errors rather than outsourcing learning tasks.

  • Examples of productive use: Effective uses include asking AI to explain topics, programming constructs, programs, and error messages.
  • Examples of productive use: Students may ask AI for analogous assignment examples or code for tasks already mastered in previous coursework.

Examples of inappropriate use

Inappropriate uses include violating assessment rules and delegating core programming work, which can undermine learning and future performance.

  • Examples of inappropriate use: Students should not use AI tools on forbidden official assessments or ask them to complete laboratory questions or assignments.
  • Examples of inappropriate use: Students should not ask AI to debug erroneous code or translate a solution into an assignment's required language.

Risks of generative AI

Generative AI can produce inaccurate, low-quality, or insecure content, while over-reliance may weaken learning, motivation, and preparation for later work.

  • Risks of generative AI: AI-generated solutions may produce correct answers while failing to use the required algorithm, so students must verify accuracy and correctness.
  • Risks of generative AI: Generated content may be generic or poor quality, and generated code may contain bugs or security flaws.
  • Risks of generative AI: Users should understand and evaluate generated code, especially because familiarity with the topic supports effective revision and assessment.
  • Risks of generative AI: Consistent over-reliance may prevent preparation for later examinations, subsequent coursework, or future job opportunities.
  • Risks of generative AI: Students need to master simple tasks before solving more complex problems, including tasks that generative AI can complete.

Impact on others

Inappropriate use of AI tools can unintentionally harm other students, burden faculty, and expose institutions to reputational or financial harm.

  • AI use can expose other students to harm by preventing their learning or compromising group-assignment academic integrity.
  • Enforcing academic-integrity standards for AI use requires faculty and administrators to spend time and energy.
  • Using AI tools without understanding their code can expose a university to reputational loss.
  • Such use can also expose the university to financial harm through lawsuits.

Academic misconduct

Using generative AI in ways that are not permitted constitutes academic misconduct and carries serious consequences.

  • Unpermitted generative-AI use is treated as academic misconduct.
  • Academic misconduct from unpermitted generative-AI use has serious consequences.
  • The policy applies specifically to generative-AI use that is not permitted.
Loading 2310.00658v1…