ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course

• ChatGPT-assisted exam answers could enable cheating and invalidate assessment results, despite models' potential to provide real-time, personalized learning support.Source: §1 · p. 1
• Prior programming-education work examined flipped classrooms, interactive video tutorials, and teacher support for students refining code.Source: §1 · pp. 1–2
• The study asks how effectively ChatGPT generates coding solutions for Java exercises in an introductory undergraduate course.Source: §1 · p. 2
• The evaluation prompts ChatGPT versions 3.5 and 4 with exercise instructions, then compares generated code with expected outcomes.Source: §1 · p. 2

• Code generation from high-level descriptions remains a longstanding challenge, and generating whole programs requires understanding tasks, reasoning about algorithms, and implementing code.Source: §2 · p. 2
• ChatGPT and AlphaCode are large language models, but ChatGPT is general-purpose while AlphaCode is specialized and trained exclusively on CodeContest datasets.Source: §2 · p. 2
• ChatGPT raises both educational concerns about cheating and interest in AI pair programming, prompting calls for assessment changes and student support.Source: §2 · p. 2
• The study explores ChatGPT's usefulness for Java exercises to inform educators' understanding of learning support and cheating-minimizing exercise design.Source: §2 · p. 2

• The undergraduate course combines pre-class videos and self-assessment quizzes with in-class coding demonstrations and asynchronous exercises.Source: §3 · p. 2
• The course exercises adhere to Oracle Java Foundations Certification requirements.Source: §3 · p. 2
• Its 80 exercises cover Java basics, object-oriented concepts, classes and methods, exceptions and I/O, and packaging.Source: §3 · p. 2

4.1 Java Basics

• For a memory-efficient string-concatenation exercise, ChatGPT generated a correct StringBuilder solution, but did not compare it with StringBuffer or the + operator.The authors note that a follow-up question could request that comparison.Source: §4.1 · p. 3
• ChatGPT could produce expected outputs for basic exercises, but ChatGPT 3.5 failed on complex output instructions that ChatGPT 4 handled correctly.The study also reports inconsistent interpretation of logical and increment/decrement operators; ChatGPT 4 answered that exercise correctly.Source: §4.1 · pp. 3–4

4.2 Object-Oriented Concepts

• Text-based object-oriented exercises requiring API documentation or class files often yielded incomplete or incorrect solutions, though ChatGPT could provide partial starting points.For the RadioActiveTest exercise, the generated method used the wrong parameters and calculation instead of the expected decay-based implementation.Source: §4.2 · p. 4

4.3 Java Classes and Methods

• When UML diagrams supplied the class and sequence details, ChatGPT produced a partially correct implementation with missing attributes and methods.The exercise instructions were text, but the diagrams were pictures.Source: §4.3 · p. 5

4.4 Java Packaging

• ChatGPT misplaced a Java source file when asked to infer its package location from the exercise and listing.It recommended sourceFiles/farming/AnimalFarmTest.java rather than sourceFiles/farming/app/AnimalFarmTest.java.Source: §4.4 · p. 5

4.5 Java Exceptions and I/O

• ChatGPT generated exception-handling code for an existing program, but struggled to determine a class’s package when that information appeared in a diagram.The example handles invalid numeric input by prompting the user to retry.Source: §4.5 · pp. 5–6

5 DISCUSSION ON THE EFFECTIVENESS OF CHATGPT

• ChatGPT can support self-directed learning through coding guidance, debugging help, and exploration of alternative or more efficient solutions.The support is especially useful outside lesson hours or in classes with high student-to-teacher ratios, while augmenting rather than replacing instructors and peers.Source: §5 · p. 6
• ChatGPT-generated solutions may be inaccurate in initial attempts, so students need to critically evaluate them using their subject knowledge.The evaluation process can contribute to students’ learning experience.Source: §5 · p. 6
• Text-based input limits ChatGPT’s effectiveness on exercises requiring interpretation of API documentation or UML diagrams.The discussion says educators can use non-text instructions to reduce direct copying, while acknowledging that advances in non-text input and prompt engineering may change this limitation.Source: §5 · p. 6
• ChatGPT-generated solutions are a promising avenue for asynchronous learning, but students’ validation of the benefit remains future work.The discussion identifies validation by students as a condition for this benefit.Source: §5 · p. 6

6 CONCLUSION

• ChatGPT-generated solutions are effective for clear, straightforward instructions, but can be partly incorrect on complex instructions and offer little benefit for API- or UML-dependent exercises.The evaluation compared generated solutions with expected outputs.Source: §6 · p. 6
• ChatGPT can help students overcome programming obstacles and learn alternative, more efficient ways to write code asynchronously.The authors also plan to integrate ChatGPT into video-based tutorials and use it to summarize StackOverflow comment assistance.Source: §6 · p. 6
• Educators should develop exercises and assessments that minimize students’ chances of copying ChatGPT solutions.Source: §6 · p. 6