PaperlayerView exact revision on arXiv

Source-linked AI summary

A GPT-based Code Review System for Programming Language Learning

Lee Dong-Kyu

arXiv:2407.04722v1cs.SEcs.AI

TL;DR

Traditional review methods have difficulty providing immediate, personalized feedback, and instructor support can be hard to scale. The paper presents a GPT-4 review system that restricts feedback to learner-submitted code and evaluates an improved version. The authors report stricter error identification, faster response, lower API costs, and no major review-quality problems.

Abstract

from arXiv · show

The increasing demand for programming language education and growing class sizes require immediate and personalized feedback. However, traditional code review methods have limitations in providing this level of feedback. As the capabilities of Large Language Models (LLMs) like GPT for generating accurate solutions and timely code reviews are verified, this research proposes a system that employs GPT-4 to offer learner-friendly code reviews and minimize the risk of AI-assist cheating. To provide learner-friendly code reviews, a dataset was collected from an online judge system, and this dataset was utilized to develop and enhance the system's prompts. In addition, to minimize AI-assist cheating, the system flow was designed to provide code reviews only for code submitted by a learner, and a feature that highlights code lines to fix was added. After the initial system was deployed on the web, software education experts conducted usability test. Based on the results, improvement strategies were developed to improve code review and code correctness check module, thereby enhancing the system. The improved system underwent evaluation by software education experts based on four criteria: strict code correctness checks, response time, lower API call costs, and the quality of code reviews. The results demonstrated a performance to accurately identify error types, shorten response times, lower API call costs, and maintain high-quality code reviews without major issues. Feedback from participants affirmed the tool's suitability for teaching programming to primary and secondary school students. Given these benefits, the system is anticipated to be a efficient learning tool in programming language learning for educational settings.

I. INTRODUCTION

The paper frames personalized, immediate code feedback as difficult to provide at scale through traditional review, while GPT offers a potential route that must address AI-assist cheating. It presents a GPT-4 system for learner-friendly reviews and evaluates an improved version against correctness, speed, cost, and review-quality questions.

A. Background

  • The study situates programming education within expanding digital-competency instruction and increased opportunities for computer and information education in Korean schools.The cited curriculum changes include expanded instructional hours in primary and secondary schools.Source: A. Background · p. 1

B. Challenge

  • Traditional code review, online test feedback, peer review, and instructor review each have constraints in delivering immediate, personalized feedback to growing classes.Online judges may be difficult for beginners to interpret, peer review takes time, and instructors face difficulty supporting many learners individually.Source: B. Challenge · p. 1

C. GPT in Programming Language Learning

  • GPT’s code-generation and analysis capabilities are presented as a potential source of timely, personalized programming feedback, alongside concerns about students using it to obtain answers.The paper argues that educational use therefore needs a system that supports code review while minimizing cheating risks.Source: C. GPT in Programming Language Learning · p. 2

D. Research Purpose

  • The system is intended for primary and secondary learners, gives reviews only on learner-submitted code, and withholds direct code solutions to minimize AI-assist cheating.Its prompts were designed around precision, usefulness, specificity, supportive tone, and learning effect.Source: D. Research Purpose · p. 2
  • The system was evaluated through four questions on stricter correctness checking, review response time, API-call cost, and review quality after usability-based improvement.The study involved two system evaluations with experienced programming-education instructors.Source: D. Research Purpose · p. 2

II. RELATED WORK

Prior work examines LLM-generated educational feedback and automated ways to explain novice programmers’ code errors. These studies highlight feedback quality and the information needed to make code-review prompts useful.

A. LLMs in Educational Feedback

  • For 103 student proposals, ChatGPT feedback was rated more detailed and readable than instructors’ feedback, while the authors caution that greater length does not necessarily mean greater effectiveness.Instructors’ feedback averaged 109 fewer characters than ChatGPT’s; three experts rated readability and consistency on a 0–4 scale.Source: A. LLMs in Educational Feedback · p. 2

B. Code Review Automation on Basic Programming training

  • Automated novice-code feedback can identify minimal corrections, but generating it requires error information and correction rules, while prompt design must address feedback composition and instructor inputs.The described feedback can identify the error line, problematic expression, sub-expression to modify, and replacement value.Source: B. Code Review Automation on Basic… · pp. 2–3

C. Prompt Template for Code Review Automation

This section describes prompt-template approaches to code review and an APR study used to inform prompt design. It highlights prompt structure, data preparation, and findings on GPT’s repair performance.

C. Prompt Template for Code Review Automation

  • The section notes that LLMs’ internalized domain knowledge can reduce the need for domain-specific prior training in code-review automation.The text presents this as a general contrast with pre-trained language models.Source: C. Prompt Template for Code Review… · p. 3
  • The cited review pipeline separates review-necessity prediction, comment generation, and code refinement into three core tasks.Comment generation may focus on individual lines or take a method-level view of code context.Source: C. Prompt Template for Code Review… · p. 3
  • The paper describes modular prompts using an instructions, optional input, and output format, which supports targeted prompt updates and maintenance.The format is based on Stanford Alpaca’s template.Source: C. Prompt Template for Code Review… · p. 3

D. The Performance of GPT on Automated Program Repair

  • APR benchmarks may overestimate LLM performance when their tasks or datasets overlap with training data, motivating evaluation on data collected without leakage.Zhang et al. collected competition problems and user-submitted code described as avoiding data leaks.Source: D. The Performance of GPT on Automated… · p. 3
  • The described APR dataset pipeline collected Java submissions, retained buggy-to-correct pairs with fewer than six changed tokens, added test cases, and filtered submissions statically and dynamically.Dynamic filtering removed pairs that failed the test cases; static filtering removed repetitive submissions and comments.Source: D. The Performance of GPT on Automated… · p. 3
  • 109 of 151 bugs were fixed with the default prompt; adding a problem description, error message, or bug location fixed 18, 25, or 10 additional bugs, respectively.Prompts containing problem or error information still left some bugs for GPT to fix; bug location helped fix 9 bugs missed by default or error-information prompts.Source: D. The Performance of GPT on Automated… · pp. 3–4
  • For GPT-based code review, removing submitted-code comments is recommended, and prompts that include the problem description, error message, and bug location can improve bug-fixing performance.The authors also note that comments in submitted code affected LLM performance during data collection and preprocessing.Source: D. The Performance of GPT on Automated… · p. 4

E. LLMs in Computer Science Class

The section presents LLMs as promising sources of programming feedback, while noting concerns about answer-generation misuse in introductory courses. It discusses prompt and interface choices intended to make feedback useful and learner-friendly.

E. LLMs in Computer Science Class

  • Instructors interviewed about LLMs in introductory programming were divided between banning classroom use to prevent cheating and introducing LLMs into classes.Prior work also found that LLMs can generate accurate solutions and explain code for introductory assignments.Source: E. LLMs in Computer Science Class · p. 4
  • Prompt guidance emphasizes examples, structured responses, technical accuracy, and a tone that does not make students uncomfortable.The text argues that prompt design and system flow should support code review while minimizing AI-assist cheating.Source: E. LLMs in Computer Science Class · p. 4
  • The proposed web interface distinguishes learner-friendly feedback from Markdown-based annotations on code lines requiring modification.The section warns that easy code generation through prompts may reduce learning effectiveness or increase learner dependence on GPT.Source: E. LLMs in Computer Science Class · p. 4

III. METHOD

The study proceeded from online-judge data collection through prompt development and system design to usability-based improvement and evaluation. Its dataset comprised filtered student submissions and instructor solutions for introductory Python exercises.

III. METHOD

  • The study followed six stages: dataset collection, prompt design, guideline establishment, initial development, usability testing and improvement, and evaluation of the improved system.The evaluation also considered suitability as a programming-learning support tool for primary and secondary students.Source: §III · p. 4

A. Dataset Collection

  • The researchers collected Python exercises, student submissions, and instructor solutions from Company C’s online judge, using a method referenced to the 2023 AtCoder Competition.The paper also cites Company C’s system as the data source.Source: A. Dataset Collection · p. 5
  • Static filtering removed repeated submissions and code comments, leaving 93 test data items for 27 questions in the dataset.The dataset table is described as an example of the resulting data frame.Source: A. Dataset Collection · p. 5, Table I · p. 5

B. Prompt Template Design and Enhancement

The authors developed a learner-oriented code-review prompt structure and revised it through repeated testing and refinement. They report reduced comment hallucination and more consistent comment style after adding solution and example sub-prompts.

B. Prompt Template Design and Enhancement

  • The prompts use the learner’s exercise, submitted code, instructor solution, and an ideal prompt example for RCGP as inputs.The paper reports five prompt-model updates, including changes to tags, wording, and prompt order.Source: B. Prompt Template Design and… · p. 5
  • The module uses role-setting, review-necessity prediction, and review-comment generation prompts, with a “no” prediction ending the response.The comment-generation prompt calls for supportive, constructive, technically accurate feedback in markdown and restricts directly presenting solutions.Source: B. Prompt Template Design and… · p. 5
  • Prompt revisions targeted clear user value, logical module structure, and efficient length while maintaining performance.Reported changes included switching from conversational to noun-style prompts and restructuring their order.Source: B. Prompt Template Design and… · p. 5
  • Adding solution and example sub-prompts reduced comment hallucinations and made comment style and tone more consistent.The Korean prompt length changed from 722 characters including spaces (553 excluding spaces) to 701 including spaces (588 excluding spaces).Source: B. Prompt Template Design and… · p. 5

C. System Design Guideline Establishment

The system guidelines combine learner-friendly feedback with mechanisms intended to localize problems and limit AI-assisted cheating. The implementation integrates the learning interface and uses GPT-4 in both code-review and correctness-check functions.

C. System Design Guideline Establishment

  • The guidelines call for immediate, accurate, supportive feedback and mark the submitted-code line needing correction so learners can locate problems.The paper frames line-level “Code to fix” comments as a response to beginners’ difficulty interpreting text error reports.Source: C. System Design Guideline Establishment · p. 6
  • To limit AI-assist cheating, the system comments only on learner-submitted code and does not provide fixed code directly.Learners request feedback through the “Ask Code Tutor” button.Source: C. System Design Guideline Establishment · p. 6
  • The guidelines also call for a single-page interface for problems, coding, and review comments, plus a Python editor intended to improve code readability and ease of writing.The implementation uses Monaco Editor for Python.Source: C. System Design Guideline Establishment · p. 6

1) System configuration and environment:

  • The web application uses Next.js for integrated backend and frontend development, with API Routes acting as a proxy for GPT endpoint requests.The paper describes two core modules: code correctness checking and code review; GPT-4 is configured to act as the compiler for the correctness check.Source: 1) System configuration and environment · pp. 6–7

2) User Interface and Execution Examples:

The interface combines exercise selection, problem details, a Python editor, code correctness checking, and GPT-4 code review. Learners can see which submitted lines need fixing and receive feedback in the system.

2) User Interface and Execution Examples:

E. Usability Test and Improvement

Three software education experts tested the initial system through hands-on use and follow-up interviews. Their feedback informed selected improvements to the code correctness and code review processes, as well as the interface and guidance for users.

E. Usability Test and Improvement

  • The usability test involved three software education experts with at least two years of programming education experience, interviewed after using the system.Interview topics covered system functions, user experience, troubleshooting, automated processes, and sentiment and opinion analysis.Source: E. Usability Test and Improvement · p. 8

1) Analyzing Usability Test Feedback:

  • Expert feedback identified correctness-check issues, including accepting only double quotation marks in strings, and prolonged grading time.The response proposed accepting varied correct answers and shortening the module’s response time.Source: 1) Analyzing Usability Test Feedback · p. 8, Table II · p. 8
  • Experts reported poor readability in problem areas and content visibility problems on small screens; proposed changes included adding color and responsive design.The design was intended to match desktop and laptop monitor resolutions.Source: 1) Analyzing Usability Test Feedback · p. 8, Table II · p. 8
  • Feedback on code review included stale comments for empty submissions, overly long responses, and inconsistent formality; proposed fixes were exception handling, concise comments, and polite wording.The exception handling was intended to prevent comments from being generated in empty-code states.Source: 1) Analyzing Usability Test Feedback · p. 8, Table II · p. 8

2) Enhancing the Initial System for Improved Usability:

IV. IMPROVED SYSTEM EVALUATION

The improved system was evaluated for stricter correctness checking and faster code-review responses. It identified error types more effectively than the existing system and reduced response times compared with the initial system.

IV. IMPROVED SYSTEM EVALUATION

  • The evaluation addressed correctness-check strictness and response time, using the existing online judge system and the initial system as respective comparators.The study also assessed API call cost and code-review quality, as described in its four research questions.Source: §IV · p. 9

A. RQ1: Strict Code Correctness Check in Improved System

  • In the online judge system, the improved system identified errors more effectively than the existing system, with particularly high detection rates for hard coding (21.3%) and unnecessary code (17.59%).Source: A. RQ1: Strict Code Correctness Check… · p. 9

B. RQ2: Response Time Reduced in Code Reviews

C. RQ3: API Call Cost Reduction via Optimization

The improved system reduced API call costs and received an overall positive code-review assessment, though some survey opinions varied. The authors conclude that it may be useful for school programming education and identify further usability work and student evaluation as future needs.

C. RQ3: API Call Cost Reduction via Optimization

D. RQ4: Quality of Code Reviews

  • Six evaluators gave an overall positive response to code-review comments, although opinions varied in some areas; participants consistently considered the tool suitable for elementary and secondary programming learning.The survey used a five-point Likert scale and evaluated precision, usefulness, specificity, supportive tone, and learning effect.Source: D. RQ4: Quality of Code Reviews · p. 10, Table VI · p. 10, Figure 14 · p. 10

V. CONCLUSION AND FURTHER STUDY

  • The authors report stricter checks for four error types, shorter response times and API costs, and no major problems with code-review quality.The error types were unnecessary code, unmet requirements, hard coding, and computation errors.Source: §V · pp. 10–11
  • Further work is needed to improve usability, introduce membership management, and verify system effectiveness with primary and secondary school students.The conclusion presents school-student effectiveness verification as follow-up research.Source: §V · p. 11
Loading 2407.04722v1…