Source-linked AI summary
Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education
Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, Mei Kuan Lim
TL;DR
The paper addresses uncertainty about whether detectors that identify AI-generated text can reliably recognize AI-generated code, a concern for educational assessment. It evaluates five detectors using varied prompts and human- and AI-generated code, and reports that several variants reduce accuracy and true negative rates for most models.
Problem
Although detectors have shown proficiency with AI-generated text, their effectiveness on code is uncertain, which may contribute to disparities in evaluating students’ submissions.
Source: §1 · p. 2Method
The study evaluates five AIGC detectors on code generated with 13 variant prompts, using datasets pairing coding problems with human- and AI-generated solutions.
Source: §1 · p. 2, §3.2 · pp. 3–4Results
Variants 2, 3, and 5 reduced accuracy and TNR across the majority of tested models, and GLTR showed variant sensitivity with accuracy ranging from 0.4841 to 0.7693.
Source: §4.2 · p. 8, §7 · p. 10Takeaways & Limitations
The findings point to a need for research to improve detector reliability and for ethical guidelines for AIGC integration in education.
Source: §7 · p. 10Code’s strict rules and distinct patterns may not exhibit the linguistic characteristics these detectors are designed to detect.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Educators are increasingly concerned about the usage of Large Language Models (LLMs) such as ChatGPT in programming education, particularly regarding the potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct. In this paper, we present an empirical study where the LLM is examined for its attempts to bypass detection by AIGC Detectors. This is achieved by generating code in response to a given question using different variants. We collected a dataset comprising 5,069 samples, with each sample consisting of a textual description of a coding problem and its corresponding human-written Python solution codes. These samples were obtained from various sources, including 80 from Quescol, 3,264 from Kaggle, and 1,725 from LeetCode. From the dataset, we created 13 sets of code problem variant prompts, which were used to instruct ChatGPT to generate the outputs. Subsequently, we assessed the performance of five AIGC detectors. Our results demonstrate that existing AIGC Detectors perform poorly in distinguishing between human-written code and AI-generated code.
1 INTRODUCTION
The introduction frames AI-generated code detection as an unresolved educational assessment problem and presents an empirical study of detectors across varied prompts. It also describes the study’s dataset and detector evaluation.
1 INTRODUCTION
- Because code-detection effectiveness is uncertain, detector-based judgments may contribute to disparities in students’ academic evaluations.The paper contrasts detectors’ proficiency with AI-generated text with uncertainty about recognizing code.Source: §1 · p. 2
- The study evaluates different AIGC Detectors across contextual and syntactical variations to assess their performance on AI-generated content.The introduction identifies this as the paper’s empirical-study aim.Source: §1 · p. 2
- The authors report evaluating five detectors with 13 variant prompts and constructing 13 datasets of 5,069 samples each, pairing problems with human- and AI-generated code.The paper describes this evaluation as, to the best of its knowledge, the first focused on detector performance across varied question prompts.Source: §1 · p. 2
2 BACKGROUND AND MOTIVATIONS
The background situates generative AI in software and computer science education, where its uses raise questions about assessment and content authenticity. It motivates examining detector vulnerabilities and identifies five detectors for study.
2 BACKGROUND AND MOTIVATIONS
- The section situates the study in software engineering and computer science education and the use of generative AI in teaching and learning.It introduces these topics as the background related to the work.Source: §2 · p. 2
2.1 Impact of AIGC on SE and CS Education
- AIGC brings learning opportunities alongside concerns about content authenticity, assessment reliability, and students potentially deceiving detectors.The paper notes that reliance on a single detector may compound concerns about automated evaluations.Source: §2.1 · p. 2
2.2 Selection of AIGC Detectors
- The study names GPTZero, Sapling, GPT-2 Detector, DetectGPT, and GLTR as detectors relevant to evaluating AIGC in software engineering education.Figure 1 outlines a workflow from generating AIGC variants through detector prediction to comparative analysis using accuracy metrics.Source: §2.2 · p. 2, Figure 1 · p. 3
3 EMPIRICAL STUDY DESIGN AND METHODOLOGY
The study evaluates AIGC detectors on AI-generated code produced from coding problems under varied prompts and code modifications. It uses 5,069 human-written solutions and 13 variants to examine detector performance and limitations.
3.3 Workflow Explanation
- The workflow collects and modifies prompts, generates and post-processes AI code, runs detectors on human- and AI-written code, then calculates performance metrics.Detection results are compiled for analysis against the study’s research questions.Source: §3 · p. 3, §3.2 · p. 4
3.1 Research Questions and Experimental Setup
- Detectors classify outputs as AI-generated when their probability exceeds 0.5; performance is assessed for accuracy, precision, TPR, FPR, TNR, and FNR.The research questions ask how accurately detectors identify AI-generated code and what limitations they have.Source: §3.1 · p. 3
3.2 Data Collection
- 5,069 human-written Python solutions came from Quescol, Kaggle, and LeetCode; prompts were used to generate AI code, with an initial collection of 70,966 generated samples.The sources contributed 80, 3,264, and 1,725 samples, respectively.Source: §3.2 · pp. 3–4
3.5 Evaluation Metrics
- Post-processing reduces some detector-specific dataset sizes: Variants 2 and 3 lose invalid ChatGPT outputs, while Sapling and DetectGPT apply minimum-word requirements.DetectGPT excludes code under 100 words; Sapling’s minimum-word requirement further reduces Variant 2.Source: §3.4 · p. 5, Table 1 · p. 6
3.4 Data Variations
- The 13 variants alter prompts or generated code, including stopword removal, human-mimic instructions, comment removal, and added assertions or test cases.These variants explore detector performance under different code and prompt conditions.Source: §3.4 · p. 5
- Other variants replace variable or function names, change method length, or add five snippets of dead code.Identifier replacements use single-character names; long and short methods are prompted as distinct code styles.Source: §3.4 · p. 5
3.5 Evaluation Metrics
- The evaluation uses a confusion matrix in which 1 denotes human-written code and 0 denotes AI-generated code, alongside accuracy, precision, recall, TPR, FPR, TNR, and FNR.F1 is described as the harmonic mean of precision and recall; AUC assesses class discrimination, with scores around 0.5 indicating random choice.Source: §3.1 · p. 3, §3.5 · pp. 5–6
4 RESULTS
Across the evaluated code variants, the five detectors often struggled to distinguish AI-generated from human-written code. Their performance also varied with prompt variants, exposing weaknesses in code detection.
4.1 RQ1: How accurate are existing AIGC Detectors at detecting AI-generated code?
- Accuracy near 0.5 for GPT-2 Detector and GPTZero, alongside high TPR and low TNR, indicates weak discrimination between AI-generated and human-written code.The paper’s RQ1 conclusion is that existing detectors perform poorly overall.Source: §4.1 · pp. 6–7, Table 3 · p. 7
- ACC near 0.5 for DetectGPT accompanies a 5% to 9% TNR improvement, while GLTR’s detection ability improves 18% to 53% over baseline variants.GLTR also shows a 6% TNR increase for Long Method (Variant 11).Source: §4.1 · pp. 6–7, Figure 3 · p. 7
- ACC above 0.6 in ten out of fourteen variants makes Sapling the strongest-performing detector reported, with TNR gains of 9% to 15% on Variants 8, 10, and 13.Those variants are Variable Name, Variable and Function Name, and DeadCode.Source: §4.1 · p. 7, Figure 4 · p. 7
4.2 RQ2: What are the limitations of existing AIGC Detectors when it comes to detecting AI-generated code?
- GLTR’s variant sensitivity is reflected in p=0.0296 and ACC ranging from 0.4841 (Variant 3) to 0.7693 (Variant 7); Variants 2, 3, and 5 lower ACC and TNR across multiple models.The paired t-test p-values for GPT-2 Detector, GPTZero, Sapling, and DetectGPT are 0.1032, 0.2479, 0.3816, and 0.5714, respectively.Source: §4.2 · pp. 7–8
- The detectors also show false positives, false negatives, and generalization challenges across code domains and when AI code resembles human coding styles.The paper notes that fine-tuning may improve performance within specific code domains without resolving those broader challenges.Source: §4.2 · p. 8
- The authors attribute code-detection weaknesses to the mismatch between programming code’s structure and patterns and detectors designed around natural-language text.They describe distinct limitations for GPTZero, Sapling, GPT-2 Detector, DetectGPT, and GLTR.Source: §4.2 · p. 8
5 DISCUSSION
The discussion recommends responsible, evidence-based integration of generative AI in SE and CS education while emphasizing that current detectors have limits for code. It also identifies prompt, dataset, and language choices that constrain the study’s validity and generalizability.
5.1 Suggestions to SE and CS Educators
- Educators should define objectives, calibrate automation, address ethics, evaluate outcomes continuously, set evidence-based policies, and stay informed about AI developments.The paper also calls for acknowledging current detector limits and further developing tools tailored to code.Source: §5.1 · p. 9
5.2 Threats to validity
- The prompts may not fully reflect real-world inputs, and ChatGPT’s non-deterministic responses can affect reproducibility.The study’s prompts aimed to simulate average user inputs.Source: §5.2.1 · p. 9
5.2.1 Internal Validity.
- Human-code authenticity is uncertain, and ambiguous coding questions can produce inaccurate AI outputs; preprocessing removed rows with such responses.The authors estimate possible LLM-generated content in public datasets had minimal impact because the datasets predated mainstream LLM use.Source: §5.2.1 · p. 9
5.2.2 Construct Validity.
- Kaggle data may overlap with ChatGPT’s training data, while conclusions may be dataset-specific and are limited to Python.The authors curated sources to broaden coverage while choosing Python to maintain consistency and control.Source: §5.2.2 · p. 10, §5.2.3 · p. 10
6 RELATED WORK
Related work frames AI-generated content detection as an academic-integrity concern, while noting the limits and risks of relying on detection tools. It also discusses broader educational benefits, ethical considerations, and policy needs.
Detection and academic integrity
- Prior work highlights limitations in AI-content detection and argues for evaluating tool accuracy and reliability alongside broader ethical approaches to academic integrity.The review also notes concerns about plagiarism involving ChatGPT-generated code [35].Source: §6 · p. 10
Detection risks
- Adnan’s comparison of generated and original abstracts reported a 1% false positive rate and warned that large-scale misclassification could reject significant research.Source: §6 · p. 10
Educational implications
- Related work reports that minority and international students face increased allegations of academic-integrity breaches from AI technologies, while AI can benefit international students and people with disabilities.It also calls for ongoing evaluation of AI-content detectors’ accuracy and reliability and ethical guidelines for responsible use in education.Source: §6 · p. 10
7 CONCLUSION AND FUTURE WORK
The study evaluates AI-content detectors on human-written and AI-generated code and reports sensitivity to code variants. It concludes that detectors face challenges from code complexity and writing-style variation, and calls for further tool development and research.
Conclusion
- The evaluation examined detectors using recall, precision, F1 score, accuracy, and AUC, and found challenges linked to code complexity and writing-style variation.The authors report that the tools hold promise for distinguishing AI-generated from human-written code.Source: §7 · p. 10
- 0.4841 to 0.7693 was the accuracy range reported for GLTR across code variants, with variants 2, 3, and 5 exacerbating discrepancies.GLTR also had a 0.0296 p-value in the Samples Paired t Test.Source: §7 · p. 10
Future Work
- Future work should improve detection across code variations and investigate long-term effects on student learning and creative engagement, as well as applicability to design artifacts.The authors also recommend responsible AI-use strategies, ethical guidelines, and ongoing detector refinement.Source: §7 · p. 10