Source-linked AI summary
Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026
Marina Lepp, Joosep Kaimre
TL;DR
Evidence about GenAI performance on authentic introductory OOP assessments remains limited, despite the growing importance of understanding these systems’ capabilities and limitations. The study evaluates five systems on original university programming tests and exam tasks, comparing their graded outputs with historical students and an earlier evaluation. The systems generally achieved high scores and improved substantially over the previous year, but recurring weaknesses remained in advanced OOP concepts and visual reasoning.
Problem
Evidence about GenAI performance on authentic OOP assessments remains limited, particularly for non-English tasks and evolving model capabilities.
Method
Five GenAI systems were evaluated on authentic Estonian Java OOP assessments using student grading criteria, historical student results, and first-response outputs without prompt engineering.
Results
The systems consistently achieved high scores, most exceeded the historical student cohort’s average, and substantial improvements appeared across nearly all systems compared with the previous year.
Takeaways & Limitations
Contemporary GenAI systems can support routine code generation, but authentic assessments still expose concepts requiring careful human verification and challenges shared with novice programmers.
Takeaways & Limitations
Persistent weaknesses included interfaces and abstract classes, graphics-related image interpretation, and the use of advanced techniques not covered in the course.
Abstract
from arXiv · showhide
Recent advances in Generative Artificial Intelligence (GenAI) have substantially improved the ability of large language models (LLMs) to generate and explain source code. However, their performance on authentic object-oriented programming (OOP) assessments remains insufficiently understood. This study evaluates five widely used GenAI systems, ChatGPT-5.2, DeepSeek-V3, Gemini 2.5 Flash, Claude Sonnet 4.5, and M365 Copilot, using programming tests and examination tasks from an introductory university OOP course. The generated solutions were assessed using the same grading criteria applied to students and compared with historical student results from the same course, as well as findings from the previous year. Common errors were also analyzed to identify recurring limitations across models. All evaluated GenAI systems achieved higher scores than the average student cohort and frequently obtained full marks on longer programming tasks. Nevertheless, they occasionally produced non-compiling code and continued to struggle with advanced OOP concepts, particularly interfaces, abstract classes, and certain inheritance-related tasks. Performance was also limited on graphics-related questions involving image interpretation. Compared with the previous year, the evaluated systems demonstrated noticeable improvements across most assessments while exhibiting several recurring error patterns. The findings provide an updated evaluation of the capabilities and limitations of contemporary GenAI systems on authentic introductory OOP assessments. They also offer evidence that can inform the design of programming assessments, the responsible integration of GenAI tools into software engineering education, and future studies evaluating the evolution of AI-assisted programming.
1. Introduction
The study addresses limited evidence about GenAI performance on authentic OOP assessments, especially in non-English and longitudinal settings. It evaluates five systems against university assessments, student results, and prior-year findings while examining recurring limitations.
- Existing evidence is insufficient because GenAI performance varies across programming topics and assessment types, and generated code may contain compilation errors or incorrect implementations.
- OOP-focused research remains limited, with only four studies identified among 125 publications on GenAI in computer science education.
- The study evaluates five GenAI systems on authentic Java OOP assessments presented in Estonian without prompt engineering.
- Historical student results provide a reference baseline, while equivalent evaluations from one year earlier support longitudinal analysis of changing capabilities.
- The paper identifies recurring conceptual and implementation errors that remain challenging despite substantial overall improvements in GenAI performance.
2. Background
Prior research finds that GenAI systems can solve many university programming tasks, but performance becomes less consistent with greater complexity. Java OOP, non-English input, advanced OOP concepts, and authentic comparisons with students remain comparatively underexplored.
- GenAI systems often perform within the top quartile of introductory programming cohorts but become less accurate as topic complexity increases.
- Non-English performance has been studied in some settings, but evidence remains mixed across task types such as exams, projects, essays, and coding snippets.
- Novice programmers commonly struggle with syntax, program behavior, abstraction, inheritance, and polymorphism, providing a comparison point for GenAI errors.
- Research on Java-based OOP courses remains limited, especially studies comparing AI assistant performance with student outcomes.
- Prior studies report difficulties with interfaces, abstract classes, class hierarchies, compilation, unit tests, and complete implementations.
3. Methodology
The study compares five GenAI systems with students using standardized programming tests and examination tasks from a Java OOP course. Identical original-language inputs, first responses, student grading criteria, and repeated assessment versions support controlled comparison.
- Research context: The study uses authentic tasks from a large first-year Java OOP course whose students generally begin without prior Java or OOP experience.
- Assessment design: Tests and the exam were selected because they are more standardized than homework and group projects, enabling comparison between AI and student performance.
- Assessment design: Programming tests require complete multi-component programs, while the exam samples conceptual understanding through varied question formats and an open-ended task.
- Systems and inputs: The five systems were selected from student-reported usage, with 87.8% of respondents reporting at least one use of AI assistants in the course.
- Systems and inputs: Systems received the original Estonian assessments with no prompt engineering, iterative refinement, or follow-up interactions, and all used default settings.
- Evaluation procedure: Only each system’s first response was graded once using the course’s standard student criteria, with non-compiling programming solutions receiving zero points.
- Evaluation procedure: Each chatbot was tested on three versions of the programming tests, while exam questions were sampled from each topic set to increase data points and reduce task-specific effects.
4. Findings
The evaluated GenAI systems performed strongly across authentic introductory OOP assessments, generally exceeding historical student averages and improving over the previous year. Their recurring weaknesses involved compilation, implementation details, advanced OOP concepts, and image-based graphics questions.
- Programming test 1: All evaluated systems except Copilot scored above the 14.61-point student average on programming test 1.Copilot's average was reduced by one non-compiling solution receiving zero points.
- Programming test 1: Programming test 1 solutions were highly accurate, with isolated compilation errors and omissions of required implementation details as the main issues.The errors generally did not reflect incorrect object-oriented structure.
- Programming test 2: Most systems achieved full or near-full scores on programming test 2, and all except Claude exceeded the 13.39-point historical student average.ChatGPT and Copilot also improved over the previous year's evaluation.
- Programming test 2: Chatbots used advanced constructs outside the course's core materials, including Comparator.comparingInt, streams, method references, and several collection methods.They also sometimes deviated from task instructions or used alternative implementations.
- Final examination: On the final examination, only DeepSeek and Copilot scored within the upper quartile, while none reached the top 10th percentile.The remaining chatbots exceeded the upper-quartile threshold but did not enter the top decile.
- Final examination: Interfaces and abstract classes remained challenging, producing misconceptions about implementation requirements, abstract methods, constants, access modifiers, and extends versus implements.Every assistant incorrectly stated that interfaces cannot contain constant variables, and all except Gemini rejected abstract methods in interfaces.
- Final examination: All systems struggled with graphics questions based on JavaFX screenshots, which were the only questions receiving zero points.Assistants often described outputs accurately but misinterpreted or could not select the corresponding image.
- Final examination: Long-form examination tasks exposed repeated failures to enumerate all valid solutions and provide complete explanations.Assistants often supplied only one valid keyword, class choice, or object-creation option when multiple answers were required.
5. Discussion
The evaluated GenAI systems achieved high scores and improved markedly over the previous year, but recurring weaknesses remained in advanced OOP concepts, implicit requirements, and visual reasoning. These error patterns indicate that strong routine code generation does not ensure reliable conceptual understanding.
- Overall performance: Most models scored above the historical student cohort, and performance improved substantially over the previous year's evaluation.ChatGPT-5.2 exceeded the upper quartile after previously scoring below the lower quartile.
- Assessment-type differences: Systems performed better on complete programming tasks than on shorter concept-oriented examination questions.The pattern suggests that conceptual reasoning remained harder than generating syntactically correct implementations, although ChatGPT was an exception.
- Recurring conceptual errors: Interfaces and abstract classes remained the most consistent sources of OOP errors.Models confused extends with implements, misunderstood @Override, and misinterpreted abstract methods and access modifiers.
- Recurring conceptual errors: Task-requirement errors included incomplete solutions, omitted implementation details, and failures to follow visibility requirements.These issues occurred even when generated code satisfied the main programming objective.
- Implementation errors: Non-compiling code and unnecessary program elements occurred occasionally despite generally functional outputs.Observed failures included syntax errors, missing imports, and unrequested classes, methods, or instance variables.
- Visual reasoning: Graphics-related questions produced lower scores because systems often misinterpreted JavaFX output or mismatched code with visual results.This limitation parallels reported difficulties with UML diagrams and other visual software-engineering artifacts.
- Educational connection: AI-generated errors clustered around abstraction, inheritance, and interfaces, resembling documented novice-programmer misconceptions.Comparing these error patterns with student misconceptions may inform programming education and future assessment design.
6. Conclusions
The study found that contemporary GenAI systems performed strongly on authentic introductory Java OOP assessments and improved substantially over the previous year. However, recurring weaknesses and the study's single-course scope limit how broadly these conclusions can be applied.
- Main findings: Most evaluated systems achieved higher average scores than the historical student cohort across programming tests and the final examination.The comparison used authentic assessments from an introductory Java-based OOP course.
- Main findings: Substantial improvements occurred across nearly all assessed systems compared with the previous year's evaluation.Repeated longitudinal evaluations are therefore valuable for tracking changing model performance.
- Persistent limitations: Interfaces, abstract classes, and graphics-related tasks continued to produce recurring errors despite overall high performance.Systems were less reliable on conceptually demanding OOP topics than on routine code generation.
- Implications: The findings support routine code-generation assistance but require careful human verification for demanding OOP tasks.Authentic assessments also identify concepts that remain difficult for both AI systems and novice programmers.
- Scope boundary: The study is limited to tasks from one Java-based OOP course, and performance may change with task phrasing or input format.Future research should examine other programming languages and paradigms.
Appendix
The appendix reports GenAI results and recurring mistakes across introductory OOP exam topics. Errors cluster around interfaces, abstract classes, inheritance, exception handling, input formats, and task-specific reasoning.
- Interfaces and abstract classes: Interfaces and abstract classes generated recurring errors, including misuse of extends, incorrect interface-variable rules, and mishandling unimplemented methods.
- Inheritance and class hierarchy: Class-hierarchy questions were answered with errors involving sorting direction, superclass access, subclass fields, and method redeclaration.
- Interfaces and abstract classes: All AI assistants incorrectly stated that interfaces cannot contain constant variables.
- Graphics and image interpretation: Image-recognition problems remained a limitation for the evaluated systems.
- Input and data structures: Input and data-structure questions exposed confusion about list indexes and incompatibility between readInt and writeUTF.
- Exceptions: Exception-handling mistakes included entering multiple catch blocks for one try statement and adding exception text to printed output.
- Conceptual OOP tasks: Longer conceptual questions showed omissions about access modifiers, interfaces, abstract classes, and using subclass instances through superclass types.